Systems and methods for operating a display system based on user perceptibility

ABSTRACT

Systems and methods are disclosed for operating a head-mounted display system based on user perceptibility. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes by presenting the content with different amounts of wavefront divergence. Some embodiments include obtaining an image captured by an imaging device of the display system. Whether a threshold measure or more of motion blur is determined to be exhibited in one or more regions of the image. Based on a determination that the threshold measure or more of motion blur is exhibited in one or more regions of the image, one or more operating parameters of the wearable display are adjusted. Example operating parameter adjustments comprise adjusting the depth plane on which content is presented (e.g., by switching from a first depth plane to a second depth plane), adjusting a rendering quality, and adjusting power characteristics of the system.

PRIORITY CLAIM

This application is a continuation of U.S. patent application Ser. No.16/389,529, entitled SYSTEMS AND METHODS FOR OPERATING A DISPLAY SYSTEMBASED ON USER PERCEPTIBILITY, filed Apr. 19, 2019; which claims priorityto U.S. Patent Prov. App. No. 62/660,180, entitled SYSTEMS AND METHODSFOR ADJUSTING OPERATIONAL PARAMETERS OF A HEAD-MOUNTED DISPLAY SYSTEMBASED ON USER SACCADES, filed Apr. 19, 2018; and U.S. Patent Prov. App.62/702,153, entitled SYSTEMS AND METHODS FOR DEPTH PLANE SWITCHING IN AMULTI-DEPTH PLANE DISPLAY SYSTEM, filed Jul. 23, 2018. Each of theabove-noted applications is incorporated herein by reference in itsentirety.

INCORPORATION BY REFERENCE

This application incorporates by reference the entirety of each of thefollowing patent applications and publications: U.S. application Ser.No. 14/555,585 filed on Nov. 27, 2014, published on Jul. 23, 2015 asU.S. Publication No. 2015/0205126; U.S. application Ser. No. 14/690,401filed on Apr. 18, 2015, published on Oct. 22, 2015 as U.S. PublicationNo. 2015/0302652; U.S. application Ser. No. 14/212,961 filed on Mar. 14,2014, now U.S. Pat. No. 9,417,452 issued on Aug. 16, 2016; U.S.application Ser. No. 14/331,218 filed on Jul. 14, 2014, published onOct. 29, 2015 as U.S. Publication No. 2015/0309263; U.S. applicationSer. No. 15/927,808 filed on Mar. 21, 2018; U.S. Application No.15/291,929 filed on Oct. 12, 2016, published on Apr. 20, 2017 as U.S.Publication No. 2017/0109580; and U.S. application Ser. No. 15/408,197filed on Jan. 17, 2017, published on Jul. 20, 2017 as U.S. PublicationNo. 2017/0206412; U.S. application Ser. No. 15/673,135, filed on Aug. 9,2017, published on Feb. 15, 2018 as U.S. Publication No. 2018/0045963;U.S. application Ser. No. 15/923,511, filed on Mar. 16, 2018, publishedon Sep. 20, 2018 as U.S. Publication No. 2018/0268220; U.S. applicationSer. No. 15/467,851, filed on Mar. 23, 2017, published on Oct. 19, 2017as U.S. Publication No. 2017/0301133; U.S. application Ser. No.15/274,823, filed on Sep. 23, 2016, published on Mar. 30, 2017 as U.S.Publication No. 2017/0091996; U.S. spplication Ser. No. 16/353,989 filedon Mar. 14, 2019.

BACKGROUND Field

The present disclosure relates to display systems, including augmentedreality imaging and visualization systems.

Description of the Related Art

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, in which digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves the presentation of digital or virtual imageinformation without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user. A mixed reality,or “MR”, scenario is a type of AR scenario and typically involvesvirtual objects that are integrated into, and responsive to, the naturalworld. For example, an MR scenario may include AR image content thatappears to be blocked by or is otherwise perceived to interact withobjects in the real world.

Referring to FIG. 1, an augmented reality scene 10 is depicted. The userof an AR technology sees a real-world park-like setting 20 featuringpeople, trees, buildings in the background, and a concrete platform 30.The user also perceives that he/she “sees” “virtual content” such as arobot statue 40 standing upon the real-world platform 30, and a flyingcartoon-like avatar character 50 which seems to be a personification ofa bumble bee. These elements 50, 40 are “virtual” in that they do notexist in the real world. Because the human visual perception system iscomplex, it is challenging to produce AR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements.

Systems and methods disclosed herein address various challenges relatedto AR and VR technology.

SUMMARY

According to some embodiments, a display system comprises a wearabledisplay configured to present virtual content to a user; an imagingdevice facing away from the wearable display, the imaging deviceconfigured to capture images; and at least one processor communicativelycoupled to the wearable display and the imaging device, the at least oneprocessor configured to: obtain an image captured by the imaging device;determine whether a threshold measure or more of motion blur isexhibited in one or more regions of the image; and in response to adetermination that the threshold measure or more of motion blur isexhibited in one or more regions of the image, adjust one or moreoperating parameters of the wearable display.

According to some embodiments, a display system comprises a wearabledisplay configured to present virtual content to a user; an imagingdevice configured to capture images of an eye of the user; at least oneprocessor communicatively coupled to the wearable display and theimaging device, the at least one processor configured to: obtain animage of the user's eye captured by the imaging device; identify aregion of the image corresponding to a target portion of the user's eye;determine whether the identified region of the image contains athreshold measure or more of motion blur; and in response to adetermination that the identified region contains the threshold measureor more, adjust one or more operating parameters of the wearabledisplay.

According to some embodiments, a display system comprises a head-mounteddisplay configured to present virtual content to a user; an imagingdevice configured to capture images of an eye of the user, the imagingdevice configured to capture images using a static frame rate andvariable exposure time; and at least one processor communicativelycoupled to the head-mounted display and the imaging device, the at leastone processor configured to: control the imaging device to dynamicallyadjust the exposure time based on one or more conditions of thehead-mounted display or the user.

According to some embodiments, a display system comprises a head-mounteddisplay configured to present virtual content to a user; an imagingdevice facing away from the head-mounted display, the imaging deviceconfigured to capture images using a static frame rate and variableexposure time; and at least one processor communicatively coupled to thehead-mounted display and the imaging device, the at least one processorconfigured to use images captured by the imaging device to perform anyof a plurality of different processes, wherein to perform each processin the plurality of different processes, the at least one processor isconfigured to: identify the process to be performed; select, from amonga plurality of different exposure times corresponding to the pluralityof different processes, respectively, an exposure time that correspondsto the identified process; control the imaging device to capture imagesusing the selected exposure time; and use the images captured by theimaging device using the selected exposure time to perform theidentified process.

According to some embodiments, a display system comprises a head-mounteddisplay configured to present virtual content to a user; two or moreimaging devices facing away from the head-mounted display, each of whichis controllable to alternate between using two different exposure timesto capture one or more images on a periodic basis; and at least oneprocessor communicatively coupled to the head-mounted display and theplurality of imaging devices, the at least one processor configured to:control the two or more imaging devices to alternate between using thetwo different exposure times out of phase with one another.

According to some embodiments, an augmented reality system comprises ahead-mounted display configured to present virtual content to a user; animaging device configured to capture images of an eye of the user; atleast one processor communicatively coupled to the head-mounted displayand the imaging device, the at least one processor configured to: obtainfirst and second consecutively-captured images of the user's eyecaptured by the imaging device; in response to obtaining the secondimage of the user's eye: identify a region of the second imagecorresponding to a target portion of the user's eye; evaluate theidentified region of the second image against a predetermined set ofcriteria; based on the evaluation of the second image, determine whetherthe second image shows the user's eye engaged in saccadic movement; andin response to a determination that the second image shows the user'seye engaged in saccadic movement, adjust one or more operatingparameters of the head-mounted display.

According to some embodiments, an augmented reality system comprises ahead-mounted display configured to present virtual content to a user; animaging device facing away from the head-mounted display, the imagingdevice configured to capture images; at least one processorcommunicatively coupled to the head-mounted display and the imagingdevice, the at least one processor configured to: obtain a first imagecaptured by the imaging device; in response to obtaining the firstimage: detect a first quantity of identifiable features in the firstimage; obtain a second image captured after the first image by theimaging device; in response to obtaining the second image: detect asecond quantity of identifiable features in the second image, the secondquantity being different from the first quantity; determine whether thesecond quantity of identifiable features is less than the first quantityof identifiable features by at least a predetermined threshold quantityof identifiable features; and in response to a determination that thesecond quantity of identifiable features is less than the first quantityof identifiable features by at least the predetermined thresholdquantity of identifiable features, adjust one or more operatingparameters of the head-mounted display.

According to some embodiments, an augmented reality display systemcomprises a head-mounted display configured to present virtual contentto a user; an imaging device configured to capture images of an eye ofthe user; at least one processor communicatively coupled to thehead-mounted display and the imaging device, the at least one processorconfigured to: obtain first and second consecutively-captured images ofthe user's eye captured by the imaging device; in response to obtainingthe first image: detect one or more eye features in the first image;based on detection of the one or more eye features in the first image,determine whether the first image contains at least a first quantity ofidentifiable eye features; in response to obtaining the second image:detect one or more eye features in the second image; based on detectionof the one or more eye features in the second image, determine whetherthe second image contains a second quantity or more of identifiable eyefeatures different from the first quantity of identifiable eye features;determine whether the second quantity of identifiable eye features isless than the first quantity of identifiable eye features by at least apredetermined threshold quantity of identifiable eye features; based atleast on a determination that the second quantity of identifiable eyefeatures is less than the first quantity of identifiable eye features byat least the predetermined threshold quantity of identifiable eyefeatures, determine that the second image shows the user's eye engagedin saccadic movement; and in response to a determination that the imageshows the user's eye engaged in saccadic movement, adjust one or moreoperating parameters of the head-mounted display.

According to some embodiments, a method may be implemented by a displaysystem comprising one or more processors, with the display systempresenting augmented reality virtual content to a user. The methodcomprises obtaining an image of an eye of the user; determining one ormore measures associated with motion blur represented in the image;determining, based on the one or more measures, that the image showsperformance of a saccade; and performing, by the display system, one ormore actions in response, the actions being associated with a reductionin visual perceptibility.

According to some embodiments, an augmented reality system comprises ahead-mounted display configured to present virtual content by outputtinglight to a user; one or more sensors; at least one processorcommunicatively coupled to the head-mounted display and the one or moresensors, the at least one processor configured to: obtain informationindicative of movement of the head-mounted display; determine one ormore measures associated with movement; in response to determining thatthe measures exceed a movement threshold, change a wavefront divergenceof the outputted light forming the virtual content.

According to some embodiments, an augmented reality system comprises ahead-mounted display configured to present virtual content by outputtinglight to a user; an imaging device configured to capture images of aneye of the user; at least one processor communicatively coupled to thehead-mounted display and the imaging device, the at least one processorconfigured to: cause presentation of virtual content, the virtualcontent being configured to be perceived as presented at one or moredepths away from the user; adjust a perceived depth of the virtualcontent; determine an accommodation vergence mismatch; and in responseto determining that the accommodation vergence mismatch exceeds athreshold, change a wavefront divergence of the outputted light formingthe virtual content.

According to some embodiments, an augmented reality system comprises ahead-mounted display configured to present virtual content by outputtinglight to a user; an imaging device configured to capture images of eyesof the user; at least one processor communicatively coupled to thehead-mounted display and the imaging device, the at least one processorconfigured to: cause presentation of the virtual content, the virtualcontent being configured to be perceived as presented at athree-dimensional location; determine a fixation point at which eyes ofthe user are fixating; determine information indicative of a differencebetween the three-dimensional location of the virtual content and thefixation point; and in response to determining that the differenceexceeds one or more thresholds, change a wavefront divergence of theoutputted light forming the virtual content.

Additional examples of embodiments are enumerated below.

Example 1. A display system comprising:

a wearable display configured to present virtual content to a user;

an imaging device facing away from the wearable display, the imagingdevice configured to capture images; and

at least one processor communicatively coupled to the wearable displayand the imaging device, the at least one processor configured to:

-   -   obtain an image captured by the imaging device;    -   determine whether a threshold measure or more of motion blur is        exhibited in one or more regions of the image; and    -   in response to a determination that the threshold measure or        more of motion blur is exhibited in one or more regions of the        image, adjust one or more operating parameters of the wearable        display.

Example 2. The display system of Example 1, wherein the imaging devicefaces the user, the imaging device configured to capture images of aneye of the user.

Example 3. The display system of Example 2, wherein to determine whetherthe threshold measure or more of motion blur is exhibited in one or moreregions of the image, the at least one processor is configured to:

identify a target region of the image corresponding to a target portionof the user's eye; and

determine whether the threshold measure or more of motion blur isexhibited in the identified region of the image.

Example 4. The display system of Example 3, further comprising aninfrared light source,

wherein the target portion of the user's eye comprises an illuminatedportion of the user's eye, wherein the illuminated portion isilluminated by the light source.

Example 5. The display system of Example 4, wherein the illuminatedportion comprises a cornea of the eye, wherein the at least oneprocessor is configured to detect infrared glint within the illuminatedportion.

Example 6. The display system of Example 4, wherein the illuminatedportion comprises a band of infrared illumination spanning a limbicboundary of the user's eye.

Example 7. The display system of Example 3, wherein the target portionof the user's eye comprises the pupil.

Example 8. The display system of Example 3, wherein the target portionof the user's eye comprises the pupil and at least a portion of an iris.

Example 9. The display system of Example 3, wherein the target portionof the user's eye comprises a portion of a limbic boundary.

Example 10. The display system of Example 1, wherein to determinewhether the threshold measure or more of motion blur is exhibited in oneor more regions of the image, the at least one processor is configuredto:

convolving at least the one or more regions with one or more imagekernels; and

determining, based on the convolution, whether the threshold measure ormore of motion blur is exhibited in the one or more regions of theimage.

Example 11. The display system of Example 1, wherein to determinewhether the threshold measure or more of motion blur is exhibited in oneor more regions of the image, the at least one processor is configuredto:

extracting frequency information from the image; and

determining, based on the frequency information, whether the thresholdmeasure or more of motion blur is exhibited in the one or more regionsof the image.

Example 12. The display system of Example 1, wherein the processor isfurther configured to determine a direction associated with motion blurexhibited in one or more regions of the image.

Example 13. The display system of Example 11, wherein the processor isfurther configured to determine a magnitude associated with motion blurexhibited in one or more regions of the image.

Example 14. The display system of Example 1, wherein the one or moreoperating parameters comprise one or more power settings of the displaysystem.

Example 15. The display system of Example 1, wherein the one or moreoperating parameters comprise one or more settings for displayingcontent on the wearable display.

Example 16. The display system of Example 1, wherein the one or moreoperating parameters comprise one or more virtual content renderingsettings.

Example 17. The display system of Example 1, wherein the at least oneprocessor is configured to change a depth plane on which virtual contentis presented in response to the determination that the identified regioncontains the threshold measure or more.

Example 18. The display system of Example 1, wherein the imaging devicefaces away from the user, the imaging device configured to captureimages of a real-world environment.

Example 19. A display system comprising:

a head-mounted display configured to present virtual content to a user;

an imaging device configured to capture images of an eye of the user,the imaging device configured to capture images using a static framerate and variable exposure time; and

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configuredto:

-   -   control the imaging device to dynamically adjust the exposure        time based on one or more conditions of the head-mounted display        or the user.

Example 20. The display system of Example 19, wherein the head-mounteddisplay is an augmented reality display comprising a stack ofwaveguides, wherein one or more of the waveguides are configured tooutput light with different amounts of wavefront divergence than othersof the waveguides.

Example 21. The display system of Example 19, wherein the at least oneprocessor is configured to control the imaging device to switch between:

(i) a short exposure time mode in which the imaging device is configuredto capture images at the static frame rate using a first exposure time;and

(ii) a long exposure time mode in which the imaging device is configuredto capture images at the static frame rate using a second exposure timethat is longer in duration than the first exposure time.

Example 22. The display system of Example 21, wherein the at least oneprocessor is configured to use images captured by the imaging device toperform one or more biometric authentication processes.

Example 23. The display system of Example 22, wherein the at least oneprocessor is configured to control the imaging device to operate in theshort exposure mode in response to initiation of one or more biometricauthentication processes.

Example 24. The display system of Example 21, wherein the at least oneprocessor is configured to use images captured by the imaging device toperform one or more saccade detection processes to determine whether theuser's eye is engaged in saccadic movement.

Example 25. The display system of Example 24, wherein the at least oneprocessor is configured to control the imaging device to operate in thelong exposure mode in response to initiation of one or more saccadedetection processes.

Example 26. The display system of Example 19, wherein to control theimaging device to dynamically adjust the exposure time based on one ormore conditions of the head-mounted display or the user, the at leastone processor is configured to:

select a particular exposure time from among a predetermined range ofexposure times based on one or more conditions of the head-mounteddisplay or the user; and

control the imaging device to shift its exposure time to the particularexposure time.

Example 27. The display system of Example 19, wherein the at least oneprocessor is further configured to:

use images captured by the imaging device to determine a fixation depthof the user, the fixation depth of the user being a depth at which eyesof the user are fixating, and

wherein to control the imaging device to dynamically adjust the exposuretime based on one or more conditions of the head-mounted display or theuser, the at least one processor is configured to:

-   -   control the imaging device to dynamically adjust the exposure        time based at least in part on the determined fixation depth of        the user.

Example 28. The display system of Example 27, wherein the at least oneprocessor is further configured to:

determine an accommodation vergence mismatch based at least in part onthe determined fixation depth of the user, and

wherein to control the imaging device to dynamically adjust the exposuretime based at least in part on the determined fixation depth of theuser, the at least one processor is configured to:

control the imaging device to dynamically adjust the exposure time basedat least in part on the determined accommodation vergence mismatch.

Example 29. The display system of Example 19, wherein the at least oneprocessor is configured to control the imaging device to alternatebetween two different exposure times on a frame-by-frame basis.

Example 30. The display system Example 29, wherein the at least oneprocessor is configured to control the imaging device to dynamicallyadjust one of the two different exposure times based on one or moreconditions of the head-mounted display or the user, while maintainingthe other of the two different exposure times at a fixed duration oftime.

Example 31. The display system of Example 29, wherein the system furthercomprises another imaging device configured to capture images of anothereye of the user.

Example 32. The display system of Example 31, wherein the at least oneprocessor is configured to:

control the other imaging device to alternate between the two differentexposure times on a frame-by-frame basis; and

control the imaging device and the other imaging device to alternatebetween the two different exposure times out of phase with one another.

Example 33. A display system comprising:

a head-mounted display configured to present virtual content to a user;

an imaging device facing away from the head-mounted display, the imagingdevice configured to capture images using a static frame rate andvariable exposure time; and

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configured touse images captured by the imaging device to perform any of a pluralityof different processes, wherein to perform each process in the pluralityof different processes, the at least one processor is configured to:

-   -   identify the process to be performed;    -   select, from among a plurality of different exposure times        corresponding to the plurality of different processes,        respectively, an exposure time that corresponds to the        identified process;    -   control the imaging device to capture images using the selected        exposure time; and    -   use the images captured by the imaging device using the selected        exposure time to perform the identified process.

Example 34. The display system of Example 33, wherein the plurality ofdifferent processes comprise a process to determine whether the user isexperiencing reduced visual perceptibility and at least one otherprocess.

Example 35. The display system of Example 34, wherein the at least oneprocessor is configured to:

use images captured by the imaging device using a first exposure time toperform the process to determine whether the user is experiencingreduced visual perceptibility; and use images captured by the imagingdevice using a second exposure time to perform the at least one otherprocess, the second exposure time being shorter in duration than thefirst exposure time.

Example 36. The display system of Example 35, wherein the imaging devicefaces the user, the imaging device configured to capture images of aneye of the user.

Example 37. The display system of Example 36, wherein to determinewhether the user is experiencing reduced visual perceptibility, the atleast one processor is configured to:

determine whether the user's eye is engaged in saccadic movement.

Example 38. The display system of Example 35, wherein the imaging devicefaces away from the user, the imaging device configured to captureimages of a real-world environment.

Example 39. The display system of Example 38, wherein to determinewhether the user is experiencing reduced visual perceptibility, the atleast one processor is configured to:

determine whether the user's head is engaged in a threshold measure ormore of movement.

Example 40. A display system comprising:

a head-mounted display configured to present virtual content to a user;two or more imaging devices facing away from the head-mounted display,each of which is controllable to alternate between using two differentexposure times to capture one or more images on a periodic basis; and

at least one processor communicatively coupled to the head-mounteddisplay and the plurality of imaging devices, the at least one processorconfigured to:

control the two or more imaging devices to alternate between using thetwo different exposure times out of phase with one another.

Example 41. The display system of Example 40, wherein the two differentexposure times comprise a first exposure time and a second exposuretime, the first exposure time being longer in duration than the secondexposure time, and wherein the at least one processor is furtherconfigured to:

use images captured by the two imaging devices using the first exposuretime to perform a first process; and

use images captured by the two imaging devices using the second exposuretime to perform a second process different from the first process.

Example 42. The display system of Example 41, wherein to use imagescaptured by the two imaging devices using the first exposure time toperform the first process, the at least one processor is configured to:

use images captured by the two imaging devices using the first exposuretime to determine whether the user is experiencing reduced visualperceptibility.

Example 43. The display system of Example 42, wherein the at least oneprocessor is further configured to:

adjust one or more operating parameters of the head-mounted display inresponse to a determination that the user is experiencing reduced visualperceptibility.

Example 44. An augmented reality system comprising:

a head-mounted display configured to present virtual content to a user;

an imaging device configured to capture images of an eye of the user;

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configuredto:

-   -   obtain first and second consecutively-captured images of the        user's eye captured by the imaging device;

in response to obtaining the second image of the user's eye:

-   -   identify a region of the second image corresponding to a target        portion of the user's eye; evaluate the identified region of the        second image against a predetermined set of criteria;

based on the evaluation of the second image, determine whether thesecond image shows the user's eye engaged in saccadic movement; and

in response to a determination that the second image shows the user'seye engaged in saccadic movement, adjust one or more operatingparameters of the head-mounted display.

Example 45. The augmented reality system of Example 44, wherein the atleast one processor is configured to:

access a kernel from a memory of the augmented reality display system;and convolve the identified region of the image with the kernel toobtain one or more measurements of motion blur contained within theidentified region of the image.

Example 46. The augmented reality system of Example 44, wherein the atleast one processor is configured to generate a confidence scoreindicating a level of confidence that the user's eyes are engaged insaccadic movement.

Example 47. The augmented reality system of Example 44, wherein the atleast one processor is configured to determine whether the second imageshows the user's eye engaged in saccadic movement based on theevaluation of the second image and irrespective of the first image.

Example 48. An augmented reality system comprising:

a head-mounted display configured to present virtual content to a user;

an imaging device facing away from the head-mounted display, the imagingdevice configured to capture images;

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configuredto:

obtain a first image captured by the imaging device;

in response to obtaining the first image:

-   -   detect a first quantity of identifiable features in the first        image; obtain a second image captured after the first image by        the imaging device; in response to obtaining the second image:    -   detect a second quantity of identifiable features in the second        image, the second quantity being different from the first        quantity;    -   determine whether the second quantity of identifiable features        is less than the first quantity of identifiable features by at        least a predetermined threshold quantity of identifiable        features; and    -   in response to a determination that the second quantity of        identifiable features is less than the first quantity of        identifiable features by at least the predetermined threshold        quantity of identifiable features, adjust one or more operating        parameters of the head-mounted display.

Example 49. The augmented reality system of Example 48, wherein theimaging device faces the user, the imaging device configured to captureimages of an eye of the user.

Example 50. The augmented reality system of Example 49, wherein theidentifiable features comprise identifiable eye features.

Example 51. The augmented reality system of Example 50, wherein the eyefeatures include one or more iris features.

Example 52. The augmented reality system of Example 51, wherein the oneor more iris features include a texture, a pattern, a key point in theiris, or a combination thereof.

Example 53. The augmented reality system of Example 50, wherein the eyefeatures include one or more scleral features.

Example 54. The augmented reality system of Example 53, wherein the oneor more scleral features include a blood vessel.

Example 55. The augmented reality system of Example 48, wherein theimaging device faces away from the user, the imaging device configuredto capture images of a real-world environment.

Example 56. The augmented reality system of Example 55, wherein theidentifiable features comprise fixed features of one or more objects inthe real-world environment.

Example 57. The augmented reality system of Example 56, wherein thefixed features comprise corners of one or more objects in the real-worldenvironment.

Example 58. The augmented reality system of Example 56, wherein thefixed features comprise edges of one or more objects in the real-worldenvironment.

Example 59. The augmented reality system of Example 48, wherein thefirst and second images comprise first and second consecutively-capturedimages.

Example 60. A method implemented by a display system comprising one ormore processors, the display system presenting augmented reality virtualcontent to a user, and the method comprising:

obtaining an image of an eye of the user;

determining one or more measures associated with motion blur representedin the image;

determining, based on the one or more measures, that the image showsperformance of a saccade; and

performing, by the display system, one or more actions in response, theactions being associated with a reduction in visual perceptibility.

Example 61. The method of Example 60, wherein determining a measureassociated with motion blur comprises:

extracting frequency information from the image; and

determining a magnitude of motion blur based on the frequencyinformation, the magnitude being utilized as the measure of motion blur.

Example 62. The method of Example 61, wherein determining a measureassociated with motion blur comprises:

convolving one or more image kernels with at least a portion of theimage; and determining the measure based on the convolution.

Example 63. The method of Example 60, wherein a plurality of measuresare determined, each measure being associated with a respective motionblur determination scheme; and

wherein determining that the image shows performance of a saccade isbased on the plurality of measures.

Example 64. The method of Example 60, wherein the measures are averaged,and wherein determining that the image shows performance of a saccadecomprises determining that the average exceeds a predetermined thresholdvalue.

Example 65. The method of Example 60, wherein determining that the imageshows performance of a saccade comprises determining that a thresholdnumber of the measures exceed respective predetermined threshold values.

Example 66. The method of Example 65, wherein the threshold number is amajority of the measures or a user-selectable threshold number.

Example 67. The method of Example 60, further comprising: determining amagnitude and direction associated with the motion blur.

Example 68. The method of Example 60, further comprising:

accessing information identifying a three-dimensional fixation pointassociated with the user, the fixation point representing a gaze of theuser; and

estimating an updated three-dimensional fixation point based on thedetermined magnitude and direction, the updated three-dimensionalfixation point representing a gaze of the user subsequent to thesaccade.

Example 69. An augmented reality system comprising:

a head-mounted display configured to present virtual content byoutputting light to a user;

one or more sensors;

at least one processor communicatively coupled to the head-mounteddisplay and the one or more sensors, the at least one processorconfigured to:

-   -   obtain information indicative of movement of the head-mounted        display;    -   determine one or more measures associated with movement;    -   in response to determining that the measures exceed a movement        threshold, change a wavefront divergence of the outputted light        forming the virtual content.

Example 70. The augmented reality system of Example 69, wherein the oneor more sensors comprise one or more accelerometers, gyroscopes,magnetometers, or a combination thereof.

Example 71. The augmented reality system of Example 69, wherein themovement threshold indicates a threshold angular velocity.

Example 72. The augmented reality system of Example 69, wherein the oneor more sensors comprise one or more outward facing cameras.

Example 73. The augmented reality system of Example 72, wherein theprocess is further configured to:

obtain, via the outward facing cameras, images of a real-worldenvironment; and determine the measures associated with movement basedon the obtained images.

Example 74. The augmented reality system of Example 73, wherein processis further configured to determine the measures by actions comprises:

monitoring movement of one or more fixed features as included in a firstimage and a subsequent, second image; or

determining measures associated with motion blur in an obtained image,the measures indicating one or more of a direction and magnitude of themotion blur.

Example 75. An augmented reality system comprising:

a head-mounted display configured to present virtual content byoutputting light to a user;

an imaging device configured to capture images of an eye of the user;

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configuredto:

-   -   cause presentation of virtual content, the virtual content being        configured to be perceived as presented at one or more depths        away from the user;    -   adjust a perceived depth of the virtual content;    -   determine an accommodation vergence mismatch; and    -   in response to determining that the accommodation vergence        mismatch exceeds a threshold, change a wavefront divergence of        the outputted light forming the virtual content.

Example 76. The augmented reality device of Example 75, whereinadjusting the perceived depth of the virtual content comprises adjustingvergence cues associated with the virtual content.

Example 77. The augmented reality device of Example 75, wherein theaccommodation vergence mismatch is based on user preference informationassociated with a user of the augmented reality device.

Example 78. The augmented reality device of Example 75, wherein thehead-mounted display is an augmented reality display comprising a stackof waveguides, wherein one or more of the waveguides are configured tooutput light with different amounts of wavefront divergence than othersof the waveguides.

Example 79. The augmented reality device of Example 78, wherein thestack of waveguides are associated with respective accommodation cues.

Example 80. The augmented reality device of Example 79, wherein thevirtual content is presented via a first waveguide associated with afirst accommodation cue, and wherein adjusting the perceived depth ofthe virtual content comprises adjusting vergence cues associated withthe virtual content.

Example 81. The augmented reality device of Example 80, whereindetermining that the accommodation-vergence mismatch exceeds a thresholdis based on a difference associated with the first accommodation cue andvergence cues exceeding the threshold.

Example 82. An augmented reality system comprising:

a head-mounted display configured to present virtual content byoutputting light to a user;

an imaging device configured to capture images of eyes of the user;

at least one processor communicatively coupled to the head-mounteddisplay and the imaging device, the at least one processor configuredto:

-   -   cause presentation of the virtual content, the virtual content        being configured to be perceived as presented at a        three-dimensional location;    -   determine a fixation point at which eyes of the user are        fixating;    -   determine information indicative of a difference between the        three-dimensional location of the virtual content and the        fixation point;    -   in response to determining that the difference exceeds one or        more thresholds, change a wavefront divergence of the outputted        light forming the virtual content.

Example 83. The augmented reality system of Example 82, whereindetermining that the difference exceeds one or more thresholds comprisesdetermining that the fixation point is greater than a threshold angulardistance from the three-dimensional location.

Example 84. The augmented reality system of Example 82, wherein theaugmented reality system is configured to use images captured by theimaging device to determine a fixation depth of the user, the fixationdepth of the user being a depth at which eyes of the user are fixating.

Example 85. The augmented reality system of Example 84, whereindetermining that the difference exceeds one or more thresholds comprisesdetermining that the fixation depth is outside of a volume ofthree-dimensional space which includes the three-dimensional location.

Example 86. The augmented reality display system of Example 82, whereinvirtual content is presented in a zone of a plurality of zones, eachzone representing a volume of real-world space, and wherein determiningthat the difference exceeds one or more thresholds is based on aparticular zone that includes the virtual content being greater than athreshold number of zones from a different zone which includes thefixation point.

Example 87. A method implemented by an augmented reality system, themethod comprising:

presenting virtual content to a user of the augmented reality system,the virtual content being configured to be perceived as presented at athree-dimensional location;

monitoring for occurrences of one or more of: an indication ofperformance of a saccade by the user, an indication of performance of ablink by the user, movement of the augmented reality system, anindication that an accommodation-vergence mismatch associated with thepresented virtual content exceeds one or more thresholds, and anindication that a difference between the three-dimensional location ofthe virtual content and a fixation point at which the user is fixatingexceeds one or more thresholds; and

based on the monitoring, changing a wavefront divergence of outputtedlight forming the virtual content.

Example 88. The method of Example 87, further comprising: determiningwhether to change the wavefront divergence based on the monitoring,wherein the determination is based on a precedence network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user's view of augmented reality (AR) through an ARdevice.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIGS. 3A-3C illustrate relationships between radius of curvature andfocal radius.

FIG. 4A illustrates a representation of the accommodation-vergenceresponse of the human visual system.

FIG. 4B illustrates examples of different accommodative states andvergence states of a pair of eyes of the user.

FIG. 4C illustrates an example of a representation of a top-down view ofa user viewing content via a display system.

FIG. 4D illustrates another example of a representation of a top-downview of a user viewing content via a display system.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user.

FIG. 7 illustrates an example of exit beams outputted by a waveguide.

FIG. 8 illustrates an example of a stacked waveguide assembly in whicheach depth plane includes images formed using multiple differentcomponent colors.

FIG. 9A illustrates a cross-sectional side view of an example of a setof stacked waveguides that each includes an incoupling optical element.

FIG. 9B illustrates a perspective view of an example of the plurality ofstacked waveguides of FIG. 9A.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B.

FIG. 9D illustrates an example of a wearable display system.

FIG. 10A illustrates example images of a user's eye as obtained by adisplay system.

FIG. 10B illustrates additional example images of a user's eye.

FIG. 11A illustrates an example process for performing display systemactions based on user saccades.

FIG. 11B illustrates another example process for performing displaysystem actions based on user saccades.

FIG. 12A illustrates a process for an example scheme to determineperformance of a saccade by a user's eye.

FIG. 12B illustrates an example block diagram of a process to determinea magnitude and direction associated with motion blur.

FIG. 13 illustrates a process for another example scheme to determineperformance of a saccade by a user's eye.

FIGS. 14A-14B illustrate another example scheme to determine performanceof a saccade based on reflected infrared light.

FIGS. 15A-15B illustrate another example scheme to determine performanceof a saccade based on reflected infrared light.

FIG. 16 illustrates an example process for depth plane switching basedon detected movement.

FIG. 17 illustrates an example process for depth plane switching basedon an accommodation-vergence mismatch.

FIG. 18 illustrates an example process for depth plane switching basedon a determined fixation point of a user.

FIG. 19 illustrates an example block diagram of the display systemconfigured to determine a time at which to perform a depth plane switch.

DETAILED DESCRIPTION

As described herein, display systems (e.g., augmented reality or virtualreality display systems) may render virtual content for presentation toa user at different perceived depths from the user. In augmented realitydisplay systems, different depth planes may be utilized to projectvirtual content with each depth plane being associated with a particularperceived depth from the user. For example, a stack of waveguidesconfigured to output light with different wavefront divergences may beutilized, with each depth plane having a corresponding wavefrontdivergence and being associated with at least one waveguide. As virtualcontent moves about the user's field of view, the virtual content may beadjusted along three discrete axes. For example, the virtual content maybe adjusted along the X, Y, and Z axes such that the virtual content maybe presented at different perceived depths from the user. The displaysystem may switch between depth planes as the virtual content isperceived to be moved further from, or closer to, the user. It will beappreciated that switching depth planes may involve changing thewavefront divergence of light forming the virtual content in a discretestep. In a waveguide-based system, in some embodiments, such a depthplane switch may involve switching the waveguide outputting light toform the virtual content. Undesirably, upon each switch, the user may beable to perceive a flicker or reduction in presentation quality as thedisplay system outputs light to the user with a discrete jump inwavefront divergence corresponding to a different depth plane.

The perceptibility of depth plane switching may be reduced by timing theswitch to coincide with various masking events, which mask or otherwisereduce the perceptibility to the user of changes in wavefront divergenceduring depth plane switching. In some embodiments, the masking eventsinclude saccades by the eyes of the user. It will be appreciated that asaccade may be a quick, simultaneous movement of both eyes that abruptlychanges the fixation point of the eyes. A saccade may reduce visualacuity or perceptibility while the saccade is being performed, forexample causing momentary saccadic blindness as the eyes move rapidly toa new fixation point. The display system may utilize the detection of asaccade as a trigger or precondition to performing a depth plane switch.Thus, the user's ability to perceive the depth-plane switch may bereduced. In some other embodiments, the performance of other displayevents or changes to displayed content may be timed to coincide with asaccade, so as to “hide” or reduce the perceptibility of the events orchanges.

Preferably, the detection and performance of display events (e.g., depthplane switching) are sufficiently quick that they occur while the useris still performing the saccade and/or before the user's visual systemreturns to perceiving visual content with high visual acuity after asaccade. Thus, early detection of a saccade may advantageously provideadditional time to perform the display events, thereby increasing thelikelihood that the display event is not perceived and/or allowinglarger numbers or types of display events to be performed while beinghidden from the user's visual perception.

For example, the display system may utilize one or more sensors (e.g.,cameras) to monitor a user's eyes. These sensors may capture information(e.g., images) at a constant, or variable, refresh rate (e.g., 30 Hz, 60Hz, and so on). With respect to the example of 30 Hz, each sampledinformation may last for 33 ms. The duration of time of a saccade mayvary, for example depending on an angular distance of eye movement.Example saccade durations include 20-200 milliseconds. Thus, for somesaccades of shorter durations, if the early portion (e.g., theinitiation) of the saccade is missed, then the performance of variousdisplay actions (e.g., depth plane switch, reduction in quality ofvirtual content) may be perceptible as they may, at least in part, beperformed after completion of the saccade. Therefore, it is advantageousto detect a saccade at an early point (e.g., an initiation of thesaccade).

Advantageously, systems and methods described herein may enabledetection of a saccade quickly and preferably prior to completion of thesaccade (e.g., at the initiation of the saccade). As noted above,saccades are characterized by quick movements of the eyes and thedisplay system may have one or more sensors that capture images of oneor both eyes. It has been found that, due to its rapid movement,individual images of an eye during the saccade may exhibit motion blur.In some embodiments, rather than tracking the movement of the eye acrossmultiple image frames, only one or two frames may be analyzed todetermine whether a saccade is occurring. As noted above, the imageframes may be captured at a particular frame rate and waiting formultiple frames to be captured undesirably consumes a portion of theduration over which a saccade may occur, thereby reducing the amount oftime available to perform a display action. On the other hand,increasing the frame rate may undesirably increase the complexity ofimaging systems and/or computing resources devoted to the image capture.Consequently, saccade detection using a single frame (or two frames) mayprovide advantages for both speed and the efficient utilization ofdisplay system resources.

In some embodiments, saccade detection schemes may utilize measures ofmotion blur of a user's eyes. As an example, a display system mayutilize sensors (e.g., cameras) to capture sensor information (e.g.,images), and determine particular measures of motion blur based on thesensor information. It will be appreciated that during a saccade both ofa person's eyes may correspondingly move, such that either eye may bemonitored. Thus, the display system may capture sensor information forone or both eyes of the user. For example, the display system may obtainan image of the user's eyes, with an exposure time (e.g., an effectiveshutter speed) set at a particular time (e.g., 5 ms, 10 ms, 20 ms).Thus, the obtained image may capture perceived movement of the user'seyes (e.g., the movement may be represented as a streak or blur in theimage). During a saccade, the perceived movement may be increased. Forexample, during a saccade the user's eyes may move at an angularvelocity between about 200 degrees per second to about 500 degrees persecond. However, during smooth pursuit (e.g., normal vision) the user'seyes may move smoothly and at a reduced angular velocity. Therefore, animage of the user's eyes may include increased motion blur uponinitiation of a saccade. This motion blur may be detected according tovarious schemes described in more detail herein. In some embodiments,the exposure duration may be chosen such that smooth pursuits do nottrigger sufficient amounts of motion blur to trigger a saccadedetection.

Example schemes to detect motion blur may include frequency spectrumtechniques, convolutional techniques, determining elongation of a user'spupil (e.g., the user's pupil may appear elongated or deformed in animage captured during a saccade), deep learning techniques (e.g., neuralnetwork-based image analysis), and so on. With respect to frequencyspectrum, a display system may perform a Fourier transform, wavelettransform, and so on, to extract frequency information from an image.The display system may then analyze the extracted frequency informationto determine magnitude and, optionally, direction associated withmovement. For example, a spatial frequency may be lower along adirection in which a user's eye is moving. With respect to convolutionaltechniques, the display system may convolve an image, or a portionthereof (e.g., a detected pupil), with one or more image kernels todetermine measures associated with motion blur.

Through use of motion blur to detect saccades, the display system maymore quickly react to such saccades. In contrast, and as will bedescribed below at least in FIG. 10C, other example schemes may utilizecomparisons between captured images to detect such events. For example,movement of specific features between successive images may bedetermined. Example features may include portions of a user's pupil, ormovement of reflected infrared lights (e.g., ‘IR’ glints) may bemonitored. For these example schemes, the display may detect a saccadeat a later point than based on analyzing motion blur included in images.For example, the display system may require a first image illustratinginitiation of a saccade, and a second image illustrating the saccade.Thus, in this example the display system may detect the saccade at alater point than based on the techniques described herein.

In some embodiments, the display system may utilize a first sensor toobtain sensor information illustrating motion blur, and a second sensorconfigured to obtain sensor information at a quicker exposure time thanthe first sensor. For example, the second sensor may obtain images of auser's eyes using a faster shutter speed, in an effort to obtain imagesof the eyes frozen (e.g., substantially frozen) in time (e.g., withoutmotion blur). In this example, the obtained images may be utilized toenhance detection of saccades by, e.g., providing a reference by whichmotion blur may be better determined. Additionally, images of both eyesmay be leveraged to improve the detection process. For example, thedisplay system may include an infrared light source configured to outputinfrared light to the user's eyes. Images obtained via the second sensormay be utilized to determine accurate geometric information associatedwith infrared light reflected from the user's eyes. Additionally, imagesobtained via the first sensor may be utilized to determine measuresassociated with motion blur. Both of these images may be leveraged toimprove saccade detection. For example, a first image may be obtainedvia the first sensor which may illustrate motion blur associated withreflected infrared light. A second image may be obtained at a same timevia the second sensor, with the second image illustrating geometricalinformation associated with the reflected infrared light (e.g., a shapeof an IR glint without deformation caused by motion blur). Comparisonsbetween the reflected infrared light between the first image and secondimage may be performed (e.g., measures of deformation and/or elongationof an IR glint), and the display system may more accurately determinemeasures of motion blur.

In some embodiments, images obtained via the second sensor may beutilized for disparate purposes, for example determiningthree-dimensional locations at which the user is fixating. For example,the first sensor may be utilized to monitor for saccades and the secondsensor may be utilized to determine locations in three-dimensional spaceat which the user is fixating. As another example, the images obtainedvia the second sensor may be utilized for biometric authenticationpurposes. For example, unique eye features of the user may be identifiedin these images and utilized to confirm an identity of the user. Exampleeye features may include iris features (e.g., a texture, pattern, keypoint in an iris, and so on), scleral features (e.g., a blood vessel),and so on. Example descriptions of eye features and identifying eyefeatures is described in more detail in U.S. Patent Pub. 2017/0109580,which is hereby incorporated by reference in its entirety for allpurposes.

In some embodiments, the display system may adjust an exposure time ofimages obtained via the first sensor and/or the second sensor. As anexample, the display system may adjust exposure time of the first sensorto obtain images illustrating motion blur, and then reduce the exposuretime to obtain images illustrating the user's eye frozen (e.g.,substantially frozen). Thus, the display system may periodically obtainimages to detect saccades, and between such detection may obtain imagesfor other purposes (e.g., determining a gaze of the eye). Similarly, thedisplay system may adjust an exposure time to perform biometricauthentication. For example, a user may perform an action to login orotherwise authenticate himself/herself, the display system may adjust anexposure time of the first sensor (e.g., temporarily) to obtain imagesfor authentication purposes.

Advantageously, as discussed herein, the ability to quickly detectsaccades allows the display system to perform various actions during thesaccade, preferably without the user being able to perceive thoseactions. An example such an action may comprise performing a switchbetween a first depth plane and a second depth plane.

As another example, the display system may reduce a resolution at whichvirtual content is being rendered during a saccade. It will beappreciated that adjusting resolution of virtual content may include anymodification to the virtual content to alter a quality of presentationof the virtual object. Such modifications may include one or more ofadjusting a polygon count of the virtual content, adjusting primitivesutilized to generate the virtual content (e.g., adjusting a shape of theprimitives, for example adjusting primitives from triangle mesh toquadrilateral mesh, and so on), adjusting operations performed on thevirtual content (e.g., shader operations), adjusting textureinformation, adjusting color resolution or depth, adjusting a number ofrendering cycles or a frame rate, and so on, including adjusting qualityat one or more points within a graphics pipeline of graphics processingunits (GPUs). Additionally, hardware characteristics may be adjusted,such as reducing a clock speed of a GPU or central processing unit(CPU).

As yet another example in an action that may be performed during asaccade, the display system may stop presentation of the virtual contentbased on detection of a saccade.

It will be appreciated that rendering virtual content may becomputationally intensive and stopping the display of virtual content orreducing the resolution of that virtual content may enable reductions inthe computational resources associated with rendering virtual content.Additionally, the power requirements of the display system may bereduced. With respect to a battery powered display system, the displaysystem's battery life may be improved and/or the display system's weightmay be reduced via a reduction in required size of the battery.

In addition to saccades, the display system may be configured to detectvarious other masking events and to perform depth plane switching upondetection of these events. As discussed herein, the depth planeswitching preferably coincides with these events. In some embodiments,the events include movement of the device, which may indicate movementof the user (e.g., movement of the user's head). The events may alsoinclude an occurrence of the user not fixating on virtual content; thedisplay system may be configured to switch depth planes when the user isdetermined to not be fixating on the virtual content. In addition, insome embodiments, even where masking events do not occur, the system maybe configured to switch depth planes upon the accommodation-vergencemismatch of virtual content exceeding a threshold, thereby preventingaccommodation-vergence mismatches that may be uncomfortable for theuser.

It will be appreciated that the display system may be part of anaugmented reality display system, or a virtual reality display system.As one example, the display of the display system may be transmissiveand may allow the user a view of the real world, while providing virtualcontent in the form of images, video, interactivity, and so on, to theuser. As another example, the display system may block the user's viewof the real world, and virtual reality images, video, interactivity, andso on, may be presented to the user.

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user. It will be appreciated that auser's eyes are spaced apart and that, when looking at a real object inspace, each eye will have a slightly different view of the object andmay form an image of the object at different locations on the retina ofeach eye. This may be referred to as binocular disparity and may beutilized by the human visual system to provide a perception of depth.Conventional display systems simulate binocular disparity by presentingtwo distinct images 190, 200 with slightly different views of the samevirtual object—one for each eye 210, 220—corresponding to the views ofthe virtual object that would be seen by each eye were the virtualobject a real object at a desired depth. These images provide binocularcues that the user's visual system may interpret to derive a perceptionof depth.

With continued reference to FIG. 2, the images 190, 200 are spaced fromthe eyes 210, 220 by a distance 230 on a z-axis. The z-axis is parallelto the optical axis of the viewer with their eyes fixated on an objectat optical infinity directly ahead of the viewer. The images 190, 200are flat and at a fixed distance from the eyes 210, 220. Based on theslightly different views of a virtual object in the images presented tothe eyes 210, 220, respectively, the eyes may naturally rotate such thatan image of the object falls on corresponding points on the retinas ofeach of the eyes, to maintain single binocular vision. This rotation maycause the lines of sight of each of the eyes 210, 220 to converge onto apoint in space at which the virtual object is perceived to be present.As a result, providing three-dimensional imagery conventionally involvesproviding binocular cues that may manipulate the vergence of the user'seyes 210, 220, and that the human visual system interprets to provide aperception of depth.

Generating a realistic and comfortable perception of depth ischallenging, however. It will be appreciated that light from objects atdifferent distances from the eyes have wavefronts with different amountsof divergence. FIGS. 3A-3C illustrate relationships between distance andthe divergence of light rays. The distance between the object and theeye 210 is represented by, in order of decreasing distance, R1, R2, andR3. As shown in FIGS. 3A-3C, the light rays become more divergent asdistance to the object decreases. Conversely, as distance increases, thelight rays become more collimated. Stated another way, it may be saidthat the light field produced by a point (the object or a part of theobject) has a spherical wavefront curvature, which is a function of howfar away the point is from the eye of the user. The curvature increaseswith decreasing distance between the object and the eye 210. While onlya single eye 210 is illustrated for clarity of illustration in FIGS.3A-3C and other figures herein, the discussions regarding eye 210 may beapplied to both eyes 210 and 220 of a viewer.

With continued reference to FIGS. 3A-3C, light from an object that theviewer's eyes are fixated on may have different degrees of wavefrontdivergence. Due to the different amounts of wavefront divergence, thelight may be focused differently by the lens of the eye, which in turnmay require the lens to assume different shapes to form a focused imageon the retina of the eye. Where a focused image is not formed on theretina, the resulting retinal blur acts as a cue to accommodation thatcauses a change in the shape of the lens of the eye until a focusedimage is formed on the retina. For example, the cue to accommodation maytrigger the ciliary muscles surrounding the lens of the eye to relax orcontract, thereby modulating the force applied to the suspensoryligaments holding the lens, thus causing the shape of the lens of theeye to change until retinal blur of an object of fixation is eliminatedor minimized, thereby forming a focused image of the object of fixationon the retina (e.g., fovea) of the eye. The process by which the lens ofthe eye changes shape may be referred to as accommodation, and the shapeof the lens of the eye required to form a focused image of the object offixation on the retina (e.g., fovea) of the eye may be referred to as anaccommodative state.

With reference now to FIG. 4A, a representation of theaccommodation-vergence response of the human visual system isillustrated. The movement of the eyes to fixate on an object causes theeyes to receive light from the object, with the light forming an imageon each of the retinas of the eyes. The presence of retinal blur in theimage formed on the retina may provide a cue to accommodation, and therelative locations of the image on the retinas may provide a cue tovergence. The cue to accommodation causes accommodation to occur,resulting in the lenses of the eyes each assuming a particularaccommodative state that forms a focused image of the object on theretina (e.g., fovea) of the eye. On the other hand, the cue to vergencecauses vergence movements (rotation of the eyes) to occur such that theimages formed on each retina of each eye are at corresponding retinalpoints that maintain single binocular vision. In these positions, theeyes may be said to have assumed a particular vergence state. Withcontinued reference to FIG. 4A, accommodation may be understood to bethe process by which the eye achieves a particular accommodative state,and vergence may be understood to be the process by which the eyeachieves a particular vergence state. As indicated in FIG. 4A, theaccommodative and vergence states of the eyes may change if the userfixates on another object. For example, the accommodated state maychange if the user fixates on a new object at a different depth on thez-axis.

Without being limited by theory, it is believed that viewers of anobject may perceive the object as being “three-dimensional” due to acombination of vergence and accommodation. As noted above, vergencemovements (e.g., rotation of the eyes so that the pupils move toward oraway from each other to converge the lines of sight of the eyes tofixate upon an object) of the two eyes relative to each other areclosely associated with accommodation of the lenses of the eyes. Undernormal conditions, changing the shapes of the lenses of the eyes tochange focus from one object to another object at a different distancewill automatically cause a matching change in vergence to the samedistance, under a relationship known as the “accommodation-vergencereflex.” Likewise, a change in vergence will trigger a matching changein lens shape under normal conditions.

With reference now to FIG. 4B, examples of different accommodative andvergence states of the eyes are illustrated. The pair of eyes 222 a arefixated on an object at optical infinity, while the pair eyes 222 b arefixated on an object 221 at less than optical infinity. Notably, thevergence states of each pair of eyes is different, with the pair of eyes222 a directed straight ahead, while the pair of eyes 222 converge onthe object 221. The accommodative states of the eyes forming each pairof eyes 222 a and 222 b are also different, as represented by thedifferent shapes of the lenses 210 a, 220 a.

Undesirably, many users of conventional “3-D” display systems find suchconventional systems to be uncomfortable or may not perceive a sense ofdepth at all due to a mismatch between accommodative and vergence statesin these displays. As noted above, many stereoscopic or “3-D” displaysystems display a scene by providing slightly different images to eacheye. Such systems are uncomfortable for many viewers, since they, amongother things, simply provide different presentations of a scene andcause changes in the vergence states of the eyes, but without acorresponding change in the accommodative states of those eyes. Rather,the images are shown by a display at a fixed distance from the eyes,such that the eyes view all the image information at a singleaccommodative state. Such an arrangement works against the“accommodation-vergence reflex” by causing changes in the vergence statewithout a matching change in the accommodative state. This mismatch isbelieved to cause viewer discomfort. Display systems that provide abetter match between accommodation and vergence may form more realisticand comfortable simulations of three-dimensional imagery.

Without being limited by theory, it is believed that the human eyetypically may interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited numbers of depthplanes. In some embodiments, the different presentations may provideboth cues to vergence and matching cues to accommodation, therebyproviding physiologically correct accommodation-vergence matching.

With continued reference to FIG. 4B, two depth planes 240, correspondingto different distances in space from the eyes 210, 220, are illustrated.For a given depth plane 240, vergence cues may be provided by thedisplaying of images of appropriately different perspectives for eacheye 210, 220. In addition, for a given depth plane 240, light formingthe images provided to each eye 210, 220 may have a wavefront divergencecorresponding to a light field produced by a point at the distance ofthat depth plane 240.

In the illustrated embodiment, the distance, along the z-axis, of thedepth plane 240 containing the point 221 is 1 m. As used herein,distances or depths along the z-axis may be measured with a zero pointlocated at the exit pupils of the user's eyes. Thus, a depth plane 240located at a depth of 1 m corresponds to a distance of 1 m away from theexit pupils of the user's eyes, on the optical axis of those eyes. As anapproximation, the depth or distance along the z-axis may be measuredfrom the display in front of the user's eyes (e.g., from the surface ofa waveguide), plus a value for the distance between the device and theexit pupils of the user's eyes, with the eyes directed towards opticalinfinity. That value may be called the eye relief and corresponds to thedistance between the exit pupil of the user's eye and the display wornby the user in front of the eye. In practice, the value for the eyerelief may be a normalized value used generally for all viewers. Forexample, the eye relief may be assumed to be 20 mm and a depth planethat is at a depth of 1 m may be at a distance of 980 mm in front of thedisplay.

With reference now to FIGS. 4C and 4D, examples of matchedaccommodation-vergence distances and mismatched accommodation-vergencedistances are illustrated, respectively. As illustrated in FIG. 4C, thedisplay system may provide images of a virtual object to each eye 210,220. The images may cause the eyes 210, 220 to assume a vergence statein which the eyes converge on a point 15 on a depth plane 240. Inaddition, the images may be formed by light having a wavefront curvaturecorresponding to real objects at that depth plane 240. As a result, theeyes 210, 220 assume an accommodative state in which the images are infocus on the retinas of those eyes. Thus, the user may perceive thevirtual object as being at the point 15 on the depth plane 240.

It will be appreciated that each of the accommodative and vergencestates of the eyes 210, 220 are associated with a particular distance onthe z-axis. For example, an object at a particular distance from theeyes 210, 220 causes those eyes to assume particular accommodativestates based upon the distances of the object. The distance associatedwith a particular accommodative state may be referred to as theaccommodation distance, Ad. Similarly, there are particular vergencedistances, Vd, associated with the eyes in particular vergence states,or positions relative to one another. Where the accommodation distanceand the vergence distance match, the relationship between accommodationand vergence may be said to be physiologically correct. This isconsidered to be the most comfortable scenario for a viewer.

In stereoscopic displays, however, the accommodation distance and thevergence distance may not always match. For example, as illustrated inFIG. 4D, images displayed to the eyes 210, 220 may be displayed withwavefront divergence corresponding to depth plane 240, and the eyes 210,220 may assume a particular accommodative state in which the points 15a, 15 b on that depth plane are in focus. However, the images displayedto the eyes 210, 220 may provide cues for vergence that cause the eyes210, 220 to converge on a point 15 that is not located on the depthplane 240. As a result, the accommodation distance corresponds to thedistance from a particular reference point of the user (e.g., the exitpupils of the eyes 210, 220) to the depth plane 240, while the vergencedistance corresponds to the larger distance from that reference point tothe point 15, in some embodiments. Thus, the accommodation distance isdifferent from the vergence distance and there is anaccommodation-vergence mismatch. Such a mismatch is consideredundesirable and may cause discomfort in the user. It will be appreciatedthat the mismatch corresponds to distance (e.g., V_(d)-A_(d)) and may becharacterized using diopters (units of reciprocal length, 1/m). Forexample, a Vd of 1.75 diopter and an A_(d) of 1.25 diopter, or a V_(d)of 1.25 diopter and an A_(d) of 1.75 diopter, would provide anaccommodation-vergence mismatch of 0.5 diopter.

In some embodiments, it will be appreciated that a reference point otherthan exit pupils of the eyes 210, 220 may be utilized for determiningdistance for determining accommodation-vergence mismatch, so long as thesame reference point is utilized for the accommodation distance and thevergence distance. For example, the distances could be measured from thecornea to the depth plane, from the retina to the depth plane, from theeyepiece (e.g., a waveguide of the display device) to the depth plane,and so on.

Without being limited by theory, it is believed that users may stillperceive accommodation-vergence mismatches of up to about 0.25 diopter,up to about 0.33 diopter, and up to about 0.5 diopter as beingphysiologically correct, without the mismatch itself causing significantdiscomfort. In some embodiments, display systems disclosed herein (e.g.,the display system 250, FIG. 6) present images to the viewer havingaccommodation-vergence mismatch of about 0.5 diopter or less. In someother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.33 diopter or less. In yetother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.25 diopter or less, includingabout 0.1 diopter or less.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence. The displaysystem includes a waveguide 270 that is configured to receive light 770that is encoded with image information, and to output that light to theuser's eye 210. The waveguide 270 may output the light 650 with adefined amount of wavefront divergence corresponding to the wavefrontdivergence of a light field produced by a point on a desired depth plane240. In some embodiments, the same amount of wavefront divergence isprovided for all objects presented on that depth plane. In addition, itwill be illustrated that the other eye of the user may be provided withimage information from a similar waveguide.

In some embodiments, a single waveguide may be configured to outputlight with a set amount of wavefront divergence corresponding to asingle or limited number of depth planes and/or the waveguide may beconfigured to output light of a limited range of wavelengths.Consequently, in some embodiments, a plurality or stack of waveguidesmay be utilized to provide different amounts of wavefront divergence fordifferent depth planes and/or to output light of different ranges ofwavelengths. As used herein, it will be appreciated at a depth plane mayfollow the contours of a flat or a curved surface. In some embodiments,for simplicity, the depth planes may follow the contours of flatsurfaces.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user. A display system 250 includes a stack ofwaveguides, or stacked waveguide assembly, 260 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 270, 280, 290, 300, 310. It will be appreciated that thedisplay system 250 may be considered a light field display in someembodiments. In addition, the waveguide assembly 260 may also bereferred to as an eyepiece.

In some embodiments, the display system 250 may be configured to providesubstantially continuous cues to vergence and multiple discrete cues toaccommodation. The cues to vergence may be provided by displayingdifferent images to each of the eyes of the user, and the cues toaccommodation may be provided by outputting the light that forms theimages with selectable discrete amounts of wavefront divergence. Statedanother way, the display system 250 may be configured to output lightwith variable levels of wavefront divergence. In some embodiments, eachdiscrete level of wavefront divergence corresponds to a particular depthplane and may be provided by a particular one of the waveguides 270,280, 290, 300, 310.

With continued reference to FIG. 6, the waveguide assembly 260 may alsoinclude a plurality of features 320, 330, 340, 350 between thewaveguides. In some embodiments, the features 320, 330, 340, 350 may beone or more lenses. The waveguides 270, 280, 290, 300, 310 and/or theplurality of lenses 320, 330, 340, 350 may be configured to send imageinformation to the eye with various levels of wavefront curvature orlight ray divergence. Each waveguide level may be associated with aparticular depth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 360, 370,380, 390, 400 may function as a source of light for the waveguides andmay be utilized to inject image information into the waveguides 270,280, 290, 300, 310, each of which may be configured, as describedherein, to distribute incoming light across each respective waveguide,for output toward the eye 210. Light exits an output surface 410, 420,430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 andis injected into a corresponding input surface 460, 470, 480, 490, 500of the waveguides 270, 280, 290, 300, 310. In some embodiments, each ofthe input surfaces 460, 470, 480, 490, 500 may be an edge of acorresponding waveguide, or may be part of a major surface of thecorresponding waveguide (that is, one of the waveguide surfaces directlyfacing the world 510 or the viewer's eye 210). In some embodiments, asingle beam of light (e.g. a collimated beam) may be injected into eachwaveguide to output an entire field of cloned collimated beams that aredirected toward the eye 210 at particular angles (and amounts ofdivergence) corresponding to the depth plane associated with aparticular waveguide. In some embodiments, a single one of the imageinjection devices 360, 370, 380, 390, 400 may be associated with andinject light into a plurality (e.g., three) of the waveguides 270, 280,290, 300, 310.

In some embodiments, the image injection devices 360, 370, 380, 390, 400are discrete displays that each produce image information for injectioninto a corresponding waveguide 270, 280, 290, 300, 310, respectively. Insome other embodiments, the image injection devices 360, 370, 380, 390,400 are the output ends of a single multiplexed display which may, e.g.,pipe image information via one or more optical conduits (such as fiberoptic cables) to each of the image injection devices 360, 370, 380, 390,400. It will be appreciated that the image information provided by theimage injection devices 360, 370, 380, 390, 400 may include light ofdifferent wavelengths, or colors (e.g., different component colors, asdiscussed herein).

In some embodiments, the light injected into the waveguides 270, 280,290, 300, 310 is provided by a light projector system 520, whichcomprises a light module 530, which may include a light emitter, such asa light emitting diode (LED). The light from the light module 530 may bedirected to and modified by a light modulator 540, e.g., a spatial lightmodulator, via a beam splitter 550. The light modulator 540 may beconfigured to change the perceived intensity of the light injected intothe waveguides 270, 280, 290, 300, 310 to encode the light with imageinformation. Examples of spatial light modulators include liquid crystaldisplays (LCD) including a liquid crystal on silicon (LCOS) displays. Itwill be appreciated that the image injection devices 360, 370, 380, 390,400 are illustrated schematically and, in some embodiments, these imageinjection devices may represent different light paths and locations in acommon projection system configured to output light into associated onesof the waveguides 270, 280, 290, 300, 310. In some embodiments, thewaveguides of the waveguide assembly 260 may function as ideal lenswhile relaying light injected into the waveguides out to the user'seyes. In this conception, the object may be the spatial light modulator540 and the image may be the image on the depth plane.

In some embodiments, the display system 250 may be a scanning fiberdisplay comprising one or more scanning fibers configured to projectlight in various patterns (e.g., raster scan, spiral scan, Lissajouspatterns, etc.) into one or more waveguides 270, 280, 290, 300, 310 andultimately to the eye 210 of the viewer. In some embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a single scanning fiber or a bundle of scanningfibers configured to inject light into one or a plurality of thewaveguides 270, 280, 290, 300, 310. In some other embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a plurality of scanning fibers or a plurality ofbundles of scanning fibers, each of which are configured to inject lightinto an associated one of the waveguides 270, 280, 290, 300, 310. Itwill be appreciated that one or more optical fibers may be configured totransmit light from the light module 530 to the one or more waveguides270, 280, 290, 300, 310. It will be appreciated that one or moreintervening optical structures may be provided between the scanningfiber, or fibers, and the one or more waveguides 270, 280, 290, 300, 310to, e.g., redirect light exiting the scanning fiber into the one or morewaveguides 270, 280, 290, 300, 310.

A controller 560 controls the operation of one or more of the stackedwaveguide assembly 260, including operation of the image injectiondevices 360, 370, 380, 390, 400, the light source 530, and the lightmodulator 540. In some embodiments, the controller 560 is part of thelocal data processing module 140. The controller 560 includesprogramming (e.g., instructions in a non-transitory medium) thatregulates the timing and provision of image information to thewaveguides 270, 280, 290, 300, 310 according to, e.g., any of thevarious schemes disclosed herein. In some embodiments, the controllermay be a single integral device, or a distributed system connected bywired or wireless communication channels. The controller 560 may be partof the processing modules 140 or 150 (FIG. 9D) in some embodiments.

With continued reference to FIG. 6, the waveguides 270, 280, 290, 300,310 may be configured to propagate light within each respectivewaveguide by total internal reflection (TIR). The waveguides 270, 280,290, 300, 310 may each be planar or have another shape (e.g., curved),with major top and bottom surfaces and edges extending between thosemajor top and bottom surfaces. In the illustrated configuration, thewaveguides 270, 280, 290, 300, 310 may each include out-coupling opticalelements 570, 580, 590, 600, 610 that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 210. Extracted light may also be referred to as out-coupledlight and the out-coupling optical elements light may also be referredto light extracting optical elements. An extracted beam of light may beoutputted by the waveguide at locations at which the light propagatingin the waveguide strikes a light extracting optical element. Theout-coupling optical elements 570, 580, 590, 600, 610 may, for example,be gratings, including diffractive optical features, as discussedfurther herein. While illustrated disposed at the bottom major surfacesof the waveguides 270, 280, 290, 300, 310, for ease of description anddrawing clarity, in some embodiments, the out-coupling optical elements570, 580, 590, 600, 610 may be disposed at the top and/or bottom majorsurfaces, and/or may be disposed directly in the volume of thewaveguides 270, 280, 290, 300, 310, as discussed further herein. In someembodiments, the out-coupling optical elements 570, 580, 590, 600, 610may be formed in a layer of material that is attached to a transparentsubstrate to form the waveguides 270, 280, 290, 300, 310. In some otherembodiments, the waveguides 270, 280, 290, 300, 310 may be a monolithicpiece of material and the out-coupling optical elements 570, 580, 590,600, 610 may be formed on a surface and/or in the interior of that pieceof material.

With continued reference to FIG. 6, as discussed herein, each waveguide270, 280, 290, 300, 310 is configured to output light to form an imagecorresponding to a particular depth plane. For example, the waveguide270 nearest the eye may be configured to deliver collimated light (whichwas injected into such waveguide 270), to the eye 210. The collimatedlight may be representative of the optical infinity focal plane. Thenext waveguide up 280 may be configured to send out collimated lightwhich passes through the first lens 350 (e.g., a negative lens) beforeit may reach the eye 210; such first lens 350 may be configured tocreate a slight convex wavefront curvature so that the eye/braininterprets light coming from that next waveguide up 280 as coming from afirst focal plane closer inward toward the eye 210 from opticalinfinity. Similarly, the third up waveguide 290 passes its output lightthrough both the first 350 and second 340 lenses before reaching the eye210; the combined optical power of the first 350 and second 340 lensesmay be configured to create another incremental amount of wavefrontcurvature so that the eye/brain interprets light coming from the thirdwaveguide 290 as coming from a second focal plane that is even closerinward toward the person from optical infinity than was light from thenext waveguide up 280.

The other waveguide layers 300, 310 and lenses 330, 320 are similarlyconfigured, with the highest waveguide 310 in the stack sending itsoutput through all of the lenses between it and the eye for an aggregatefocal power representative of the closest focal plane to the person. Tocompensate for the stack of lenses 320, 330, 340, 350 whenviewing/interpreting light coming from the world 510 on the other sideof the stacked waveguide assembly 260, a compensating lens layer 620 maybe disposed at the top of the stack to compensate for the aggregatepower of the lens stack 320, 330, 340, 350 below. Such a configurationprovides as many perceived focal planes as there are availablewaveguide/lens pairings. Both the out-coupling optical elements of thewaveguides and the focusing aspects of the lenses may be static (i.e.,not dynamic or electro-active). In some alternative embodiments, eitheror both may be dynamic using electro-active features.

In some embodiments, two or more of the waveguides 270, 280, 290, 300,310 may have the same associated depth plane. For example, multiplewaveguides 270, 280, 290, 300, 310 may be configured to output imagesset to the same depth plane, or multiple subsets of the waveguides 270,280, 290, 300, 310 may be configured to output images set to the sameplurality of depth planes, with one set for each depth plane. This mayprovide advantages for forming a tiled image to provide an expandedfield of view at those depth planes.

With continued reference to FIG. 6, the out-coupling optical elements570, 580, 590, 600, 610 may be configured to both redirect light out oftheir respective waveguides and to output this light with theappropriate amount of divergence or collimation for a particular depthplane associated with the waveguide. As a result, waveguides havingdifferent associated depth planes may have different configurations ofout-coupling optical elements 570, 580, 590, 600, 610, which outputlight with a different amount of divergence depending on the associateddepth plane. In some embodiments, the light extracting optical elements570, 580, 590, 600, 610 may be volumetric or surface features, which maybe configured to output light at specific angles. For example, the lightextracting optical elements 570, 580, 590, 600, 610 may be volumeholograms, surface holograms, and/or diffraction gratings. In someembodiments, the features 320, 330, 340, 350 may not be lenses; rather,they may simply be spacers (e.g., cladding layers and/or structures forforming air gaps).

In some embodiments, the out-coupling optical elements 570, 580, 590,600, 610 are diffractive features that form a diffraction pattern, or“diffractive optical element” (also referred to herein as a “DOE”).Preferably, the DOE's have a sufficiently low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye 210 with each intersection of the DOE, while the rest continuesto move through a waveguide via TIR. The light carrying the imageinformation is thus divided into a number of related exit beams thatexit the waveguide at a multiplicity of locations and the result is afairly uniform pattern of exit emission toward the eye 210 for thisparticular collimated beam bouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets may be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet may be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, a camera assembly 630 (e.g., a digital camera,including visible light and infrared light cameras) may be provided tocapture images of the eye 210 and/or tissue around the eye 210 to, e.g.,detect user inputs and/or to monitor the physiological state of theuser. As used herein, a camera may be any image capture device. In someembodiments, the camera assembly 630 may include an image capture deviceand a light source to project light (e.g., infrared light) to the eye,which may then be reflected by the eye and detected by the image capturedevice. In some embodiments, the camera assembly 630 may be attached tothe frame 80 (FIG. 9D) and may be in electrical communication with theprocessing modules 140 and/or 150, which may process image informationfrom the camera assembly 630. In some embodiments, one camera assembly630 may be utilized for each eye, to separately monitor each eye.

With reference now to FIG. 7, an example of exit beams outputted by awaveguide is shown. One waveguide is illustrated, but it will beappreciated that other waveguides in the waveguide assembly 260 (FIG. 6)may function similarly, where the waveguide assembly 260 includesmultiple waveguides. Light 640 is injected into the waveguide 270 at theinput surface 460 of the waveguide 270 and propagates within thewaveguide 270 by TIR. At points where the light 640 impinges on the DOE570, a portion of the light exits the waveguide as exit beams 650. Theexit beams 650 are illustrated as substantially parallel but, asdiscussed herein, they may also be redirected to propagate to the eye210 at an angle (e.g., forming divergent exit beams), depending on thedepth plane associated with the waveguide 270. It will be appreciatedthat substantially parallel exit beams may be indicative of a waveguidewith out-coupling optical elements that out-couple light to form imagesthat appear to be set on a depth plane at a large distance (e.g.,optical infinity) from the eye 210. Other waveguides or other sets ofout-coupling optical elements may output an exit beam pattern that ismore divergent, which would require the eye 210 to accommodate to acloser distance to bring it into focus on the retina and would beinterpreted by the brain as light from a distance closer to the eye 210than optical infinity.

In some embodiments, a full color image may be formed at each depthplane by overlaying images in each of the component colors, e.g., threeor more component colors. FIG. 8 illustrates an example of a stackedwaveguide assembly in which each depth plane includes images formedusing multiple different component colors. The illustrated embodimentshows depth planes 240 a-240 f, although more or fewer depths are alsocontemplated. Each depth plane may have three or more component colorimages associated with it, including: a first image of a first color, G;a second image of a second color, R; and a third image of a third color,B. Different depth planes are indicated in the figure by differentnumbers for diopters (dpt) following the letters G, R, and B. Just asexamples, the numbers following each of these letters indicate diopters(1/m), or inverse distance of the depth plane from a viewer, and eachbox in the figures represents an individual component color image. Insome embodiments, to account for differences in the eye's focusing oflight of different wavelengths, the exact placement of the depth planesfor different component colors may vary. For example, differentcomponent color images for a given depth plane may be placed on depthplanes corresponding to different distances from the user. Such anarrangement may increase visual acuity and user comfort and/or maydecrease chromatic aberrations.

In some embodiments, light of each component color may be outputted by asingle dedicated waveguide and, consequently, each depth plane may havemultiple waveguides associated with it. In such embodiments, each box inthe figures including the letters G, R, or B may be understood torepresent an individual waveguide, and three waveguides may be providedper depth plane where three component color images are provided perdepth plane. While the waveguides associated with each depth plane areshown adjacent to one another in this drawing for ease of description,it will be appreciated that, in a physical device, the waveguides mayall be arranged in a stack with one waveguide per level. In some otherembodiments, multiple component colors may be outputted by the samewaveguide, such that, e.g., only a single waveguide may be provided perdepth plane.

With continued reference to FIG. 8, in some embodiments, G is the colorgreen, R is the color red, and B is the color blue. In some otherembodiments, other colors associated with other wavelengths of light,including magenta and cyan, may be used in addition to or may replaceone or more of red, green, or blue.

It will be appreciated that references to a given color of lightthroughout this disclosure will be understood to encompass light of oneor more wavelengths within a range of wavelengths of light that areperceived by a viewer as being of that given color. For example, redlight may include light of one or more wavelengths in the range of about620-780 nm, green light may include light of one or more wavelengths inthe range of about 492-577 nm, and blue light may include light of oneor more wavelengths in the range of about 435-493 nm.

In some embodiments, the light source 530 (FIG. 6) may be configured toemit light of one or more wavelengths outside the visual perceptionrange of the viewer, for example, infrared and/or ultravioletwavelengths. In addition, the in-coupling, out-coupling, and other lightredirecting structures of the waveguides of the display 250 may beconfigured to direct and emit this light out of the display towards theuser's eye 210, e.g., for imaging and/or user stimulation applications.

With reference now to FIG. 9A, in some embodiments, light impinging on awaveguide may need to be redirected to in-couple that light into thewaveguide. An in-coupling optical element may be used to redirect andin-couple the light into its corresponding waveguide. FIG. 9Aillustrates a cross-sectional side view of an example of a plurality orset 660 of stacked waveguides that each includes an in-coupling opticalelement. The waveguides may each be configured to output light of one ormore different wavelengths, or one or more different ranges ofwavelengths. It will be appreciated that the stack 660 may correspond tothe stack 260 (FIG. 6) and the illustrated waveguides of the stack 660may correspond to part of the plurality of waveguides 270, 280, 290,300, 310, except that light from one or more of the image injectiondevices 360, 370, 380, 390, 400 is injected into the waveguides from aposition that requires light to be redirected for in-coupling.

The illustrated set 660 of stacked waveguides includes waveguides 670,680, and 690. Each waveguide includes an associated in-coupling opticalelement (which may also be referred to as a light input area on thewaveguide), with, e.g., in-coupling optical element 700 disposed on amajor surface (e.g., an upper major surface) of waveguide 670,in-coupling optical element 710 disposed on a major surface (e.g., anupper major surface) of waveguide 680, and in-coupling optical element720 disposed on a major surface (e.g., an upper major surface) ofwaveguide 690. In some embodiments, one or more of the in-couplingoptical elements 700, 710, 720 may be disposed on the bottom majorsurface of the respective waveguide 670, 680, 690 (particularly wherethe one or more in-coupling optical elements are reflective, deflectingoptical elements). As illustrated, the in-coupling optical elements 700,710, 720 may be disposed on the upper major surface of their respectivewaveguide 670, 680, 690 (or the top of the next lower waveguide),particularly where those in-coupling optical elements are transmissive,deflecting optical elements. In some embodiments, the in-couplingoptical elements 700, 710, 720 may be disposed in the body of therespective waveguide 670, 680, 690. In some embodiments, as discussedherein, the in-coupling optical elements 700, 710, 720 are wavelengthselective, such that they selectively redirect one or more wavelengthsof light, while transmitting other wavelengths of light. Whileillustrated on one side or corner of their respective waveguide 670,680, 690, it will be appreciated that the in-coupling optical elements700, 710, 720 may be disposed in other areas of their respectivewaveguide 670, 680, 690 in some embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 6, and may be separated (e.g., laterallyspaced apart) from other in-coupling optical elements 700, 710, 720 suchthat it substantially does not receive light from the other ones of thein-coupling optical elements 700, 710, 720.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750, may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750, may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690, respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, and/or solid layers of material. For example, asillustrated, layer 760 a may separate waveguides 670 and 680; and layer760 b may separate waveguides 680 and 690. In some embodiments, thelayers 760 a and 760 b are formed of low refractive index materials (forexample, materials having a lower refractive index than the materialforming the immediately adjacent one of waveguides 670, 680, 690).Preferably, the refractive index of the material forming the layers 760a, 760 b is 0.05 or more, or 0.10 or less than the refractive index ofthe material forming the waveguides 670, 680, 690. Advantageously, thelower refractive index layers 760 a, 760 b may function as claddinglayers that facilitate total internal reflection (TIR) of light throughthe waveguides 670, 680, 690 (e.g., TIR between the top and bottom majorsurfaces of each waveguide). In some embodiments, the layers 760 a, 760b are formed of air. While not illustrated, it will be appreciated thatthe top and bottom of the illustrated set 660 of waveguides may includeimmediately neighboring cladding layers.

Preferably, for ease of manufacturing and other considerations, thematerial forming the waveguides 670, 680, 690 are similar or the same,and the material forming the layers 760 a, 760 b are similar or thesame. In some embodiments, the material forming the waveguides 670, 680,690 may be different between one or more waveguides, and/or the materialforming the layers 760 a, 760 b may be different, while still holding tothe various refractive index relationships noted above.

With continued reference to FIG. 9A, light rays 770, 780, 790 areincident on the set 660 of waveguides. It will be appreciated that thelight rays 770, 780, 790 may be injected into the waveguides 670, 680,690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG.6).

In some embodiments, the light rays 770, 780, 790 have differentproperties, e.g., different wavelengths or different ranges ofwavelengths, which may correspond to different colors. The in-couplingoptical elements 700, 710, 720 each deflect the incident light such thatthe light propagates through a respective one of the waveguides 670,680, 690 by TIR. In some embodiments, the incoupling optical elements700, 710, 720 each selectively deflect one or more particularwavelengths of light, while transmitting other wavelengths to anunderlying waveguide and associated incoupling optical element.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths,while transmitting rays 780 and 790, which have different second andthird wavelengths or ranges of wavelengths, respectively. Thetransmitted ray 780 impinges on and is deflected by the in-couplingoptical element 710, which is configured to deflect light of a secondwavelength or range of wavelengths. The ray 790 is deflected by thein-coupling optical element 720, which is configured to selectivelydeflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 9A, the deflected light rays 770, 780,790 are deflected so that they propagate through a correspondingwaveguide 670, 680, 690; for example, the in-coupling optical elements700, 710, 720 of each waveguide deflects light into that correspondingwaveguide 670, 680, 690 to in-couple light into that correspondingwaveguide. The light rays 770, 780, 790 are deflected at angles thatcause the light to propagate through the respective waveguide 670, 680,690 by TIR. The light rays 770, 780, 790 propagate through therespective waveguide 670, 680, 690 by TIR until impinging on thewaveguide's corresponding light distributing elements 730, 740, 750.

With reference now to FIG. 9B, a perspective view of an example of theplurality of stacked waveguides of FIG. 9A is illustrated. As notedabove, the in-coupled light rays 770, 780, 790, are deflected by thein-coupling optical elements 700, 710, 720, respectively, and thenpropagate by TIR within the waveguides 670, 680, 690, respectively. Thelight rays 770, 780, 790 then impinge on the light distributing elements730, 740, 750, respectively. The light distributing elements 730, 740,750 deflect the light rays 770, 780, 790 so that they propagate towardsthe out-coupling optical elements 800, 810, 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE's). In some embodiments, the OPE'sdeflect or distribute light to the out-coupling optical elements 800,810, 820 and, in some embodiments, may also increase the beam or spotsize of this light as it propagates to the out-coupling opticalelements. In some embodiments, the light distributing elements 730, 740,750 may be omitted and the in-coupling optical elements 700, 710, 720may be configured to deflect light directly to the out-coupling opticalelements 800, 810, 820. For example, with reference to FIG. 9A, thelight distributing elements 730, 740, 750 may be replaced without-coupling optical elements 800, 810, 820, respectively. In someembodiments, the out-coupling optical elements 800, 810, 820 are exitpupils (EP's) or exit pupil expanders (EPE's) that direct light in aviewer's eye 210 (FIG. 7). It will be appreciated that the OPE's may beconfigured to increase the dimensions of the eye box in at least oneaxis and the EPE's may be to increase the eye box in an axis crossing,e.g., orthogonal to, the axis of the OPEs. For example, each OPE may beconfigured to redirect a portion of the light striking the OPE to an EPEof the same waveguide, while allowing the remaining portion of the lightto continue to propagate down the waveguide. Upon impinging on the OPEagain, another portion of the remaining light is redirected to the EPE,and the remaining portion of that portion continues to propagate furtherdown the waveguide, and so on. Similarly, upon striking the EPE, aportion of the impinging light is directed out of the waveguide towardsthe user, and a remaining portion of that light continues to propagatethrough the waveguide until it strikes the EP again, at which timeanother portion of the impinging light is directed out of the waveguide,and so on. Consequently, a single beam of incoupled light may be“replicated” each time a portion of that light is redirected by an OPEor EPE, thereby forming a field of cloned beams of light, as shown inFIG. 6. In some embodiments, the OPE and/or EPE may be configured tomodify a size of the beams of light.

Accordingly, with reference to FIGS. 9A and 9B, in some embodiments, theset 660 of waveguides includes waveguides 670, 680, 690; in-couplingoptical elements 700, 710, 720; light distributing elements (e.g.,OPE's) 730, 740, 750; and out-coupling optical elements (e.g., EP's)800, 810, 820 for each component color. The waveguides 670, 680, 690 maybe stacked with an air gap/cladding layer between each one. Thein-coupling optical elements 700, 710, 720 redirect or deflect incidentlight (with different in-coupling optical elements receiving light ofdifferent wavelengths) into its waveguide. The light then propagates atan angle which will result in TIR within the respective waveguide 670,680, 690. In the example shown, light ray 770 (e.g., blue light) isdeflected by the first in-coupling optical element 700, and thencontinues to bounce down the waveguide, interacting with the lightdistributing element (e.g., OPE's) 730 and then the out-coupling opticalelement (e.g., EPs) 800, in a manner described earlier. The light rays780 and 790 (e.g., green and red light, respectively) will pass throughthe waveguide 670, with light ray 780 impinging on and being deflectedby in-coupling optical element 710. The light ray 780 then bounces downthe waveguide 680 via TIR, proceeding on to its light distributingelement (e.g., OPEs) 740 and then the out-coupling optical element(e.g., EP's) 810. Finally, light ray 790 (e.g., red light) passesthrough the waveguide 690 to impinge on the light in-coupling opticalelements 720 of the waveguide 690. The light in-coupling opticalelements 720 deflect the light ray 790 such that the light raypropagates to light distributing element (e.g., OPEs) 750 by TIR, andthen to the out-coupling optical element (e.g., EPs) 820 by TIR. Theout-coupling optical element 820 then finally out-couples the light ray790 to the viewer, who also receives the out-coupled light from theother waveguides 670, 680.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B. As illustrated, the waveguides670, 680, 690, along with each waveguide's associated light distributingelement 730, 740, 750 and associated out-coupling optical element 800,810, 820, may be vertically aligned. However, as discussed herein, thein-coupling optical elements 700, 710, 720 are not vertically aligned;rather, the in-coupling optical elements are preferably non-overlapping(e.g., laterally spaced apart as seen in the top-down view). Asdiscussed further herein, this nonoverlapping spatial arrangementfacilitates the injection of light from different resources intodifferent waveguides on a one-to-one basis, thereby allowing a specificlight source to be uniquely coupled to a specific waveguide. In someembodiments, arrangements including nonoverlapping spatially-separatedin-coupling optical elements may be referred to as a shifted pupilsystem, and the in-coupling optical elements within these arrangementsmay correspond to sub pupils.

FIG. 9D illustrates an example of wearable display system 60 into whichthe various waveguides and related systems disclosed herein may beintegrated. In some embodiments, the display system 60 is the system 250of FIG. 6, with FIG. 6 schematically showing some parts of that system60 in greater detail. For example, the waveguide assembly 260 of FIG. 6may be part of the display 70.

With continued reference to FIG. 9D, the display system 60 includes adisplay 70, and various mechanical and electronic modules and systems tosupport the functioning of that display 70. The display 70 may becoupled to a frame 80, which is wearable by a display system user orviewer 90 and which is configured to position the display 70 in front ofthe eyes of the user 90. The display 70 may be considered eyewear insome embodiments. In some embodiments, a speaker 100 is coupled to theframe 80 and configured to be positioned adjacent the ear canal of theuser 90 (in some embodiments, another speaker, not shown, may optionallybe positioned adjacent the other ear canal of the user to providestereo/shapeable sound control). The display system 60 may also includeone or more microphones 110 or other devices to detect sound. In someembodiments, the microphone is configured to allow the user to provideinputs or commands to the system 60 (e.g., the selection of voice menucommands, natural language questions, etc.), and/or may allow audiocommunication with other persons (e.g., with other users of similardisplay systems. The microphone may further be configured as aperipheral sensor to collect audio data (e.g., sounds from the userand/or environment). In some embodiments, the display system 60 mayfurther include one or more outwardly-directed environmental sensors 112configured to detect light, objects, stimuli, people, animals,locations, or other aspects of the world around the user. For example,environmental sensors 112 may include one or more cameras, which may belocated, for example, facing outward so as to capture images similar toat least a portion of an ordinary field of view of the user 90. In someembodiments, the display system may also include a peripheral sensor 120a, which may be separate from the frame 80 and attached to the body ofthe user 90 (e.g., on the head, torso, an extremity, etc. of the user90). The peripheral sensor 120 a may be configured to acquire datacharacterizing a physiological state of the user 90 in some embodiments.For example, the sensor 120 a may be an electrode.

With continued reference to FIG. 9D, the display 70 is operativelycoupled by communications link 130, such as by a wired lead or wirelessconnectivity, to a local data processing module 140 which may be mountedin a variety of configurations, such as fixedly attached to the frame80, fixedly attached to a helmet or hat worn by the user, embedded inheadphones, or otherwise removably attached to the user 90 (e.g., in abackpack-style configuration, in a belt-coupling style configuration).Similarly, the sensor 120 a may be operatively coupled by communicationslink 120 b, e.g., a wired lead or wireless connectivity, to the localprocessor and data module 140. The local processing and data module 140may comprise a hardware processor, as well as digital memory, such asnon-volatile memory (e.g., flash memory or hard disk drives), both ofwhich may be utilized to assist in the processing, caching, and storageof data. Optionally, the local processor and data module 140 may includeone or more central processing units (CPUs), graphics processing units(GPUs), dedicated processing hardware, and so on. The data may includedata a) captured from sensors (which may be, e.g., operatively coupledto the frame 80 or otherwise attached to the user 90), such as imagecapture devices (such as cameras), microphones, inertial measurementunits, accelerometers, compasses, GPS units, radio devices, gyros,and/or other sensors disclosed herein; and/or b) acquired and/orprocessed using remote processing module 150 and/or remote datarepository 160 (including data relating to virtual content), possiblyfor passage to the display 70 after such processing or retrieval. Thelocal processing and data module 140 may be operatively coupled bycommunication links 170, 180, such as via a wired or wirelesscommunication links, to the remote processing module 150 and remote datarepository 160 such that these remote modules 150, 160 are operativelycoupled to each other and available as resources to the local processingand data module 140. In some embodiments, the local processing and datamodule 140 may include one or more of the image capture devices,microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, and/or gyros. In some other embodiments, one ormore of these sensors may be attached to the frame 80, or may bestandalone structures that communicate with the local processing anddata module 140 by wired or wireless communication pathways.

With continued reference to FIG. 9D, in some embodiments, the remoteprocessing module 150 may comprise one or more processors configured toanalyze and process data and/or image information, for instanceincluding one or more central processing units (CPUs), graphicsprocessing units (GPUs), dedicated processing hardware, and so on. Insome embodiments, the remote data repository 160 may comprise a digitaldata storage facility, which may be available through the internet orother networking configuration in a “cloud” resource configuration. Insome embodiments, the remote data repository 160 may include one or moreremote servers, which provide information, e.g., information forgenerating augmented reality content, to the local processing and datamodule 140 and/or the remote processing module 150. In some embodiments,all data is stored and all computations are performed in the localprocessing and data module, allowing fully autonomous use from a remotemodule.

Optionally, an outside system (e.g., a system of one or more processors,one or more computers) that includes CPUs, GPUs, and so on, may performat least a portion of processing (e.g., generating image information,processing data) and provide information to, and receive informationfrom, modules 140, 150, 160, for instance via wireless or wiredconnections.

Detection of Saccades

As described above, a display system (e.g., display system 60) may beconfigured to detect saccades during which a user advantageously hasreduced visual perceptibility. In response, the display system may beconfigured to advantageously perform one or more actions to, e.g., (1)improve user experience and/or (2) reduce required computationalresources or power usage. For example, the display system may detect anoccurrence of a saccade and perform one or more display actions whilethe user's visual acuity or perceptibility is reduced, therebyeffectively providing saccadic masking of the action. Example actions toimprove user experience may include switching presentation of virtualcontent to the user from a first depth plane to a second depth plane,while masking the perceptibility of this switch. As described above, thedisplay system may include a stack of waveguides that are configured tooutput light with different wavefront divergence. These wavefrontdivergences may correspond to discrete perceived depths from the user.As virtual content moves within the user's field of view (e.g., within adisplay frustum), the virtual content may be adjusted in perceived depthfrom the user. Thus, as virtual content exits depths covered by a firstdepth plane (e.g., a range of depths, such as measured in diopter), andenters depths covered by a second depth plane, the display system mayswitch between waveguides. This switching may introduce perceptibleflicker to the user. In some embodiments, the display system may mask orhide this switching by performing the switch while the eyes areperforming a saccade.

In some embodiments, the display system may reduce the usage ofcomputational resources or power based on detection of saccades. In someembodiments, the reduction may occur for a duration of the saccade. Forexample, the display system may adjust a virtual content renderingsetting (e.g., reduce a resolution at which virtual content is beingrendered), a display setting (e.g., reduce a refresh rate of thepresented virtual content, a brightness, contrast ratio, color settings,and so on), a power setting (e.g., clock speed of the CPU and/or GPU maybe temporarily reduced, the display may be turned off, and so on). Inthe example of reducing resolution, the display system may reduce apolygon count associated with the virtual content, reduce textureinformation presented on the polygons, reduce lighting or otherpost-processing effects, and so on. As another example, the display mayturn off presentation of virtual content.

In some embodiments, when reducing resolution, the display system mayobtain information identifying a proximity of the virtual content to agaze of the user. For example, the information may indicate a proximityof virtual content to a three-dimensional fixation point of the user. Itwill be appreciated that a visual acuity or perceptibility of a user maybe reduced based on a distance (e.g., angular distance,three-dimensional distance) from a location at which the user isfixating. For example, visual acuity may be greatest at an immediatelyproximate a fixation point, and decrease in three dimensions away fromthe fixation point. Thus, in some embodiments, the display system mayrender virtual content at different resolutions based on respectivedistance of the virtual content from the fixation point. During asaccade, the display system may further reduce resolution of the virtualcontent. For particular virtual content located at greater than athreshold distance (e.g., angular distance) from the fixation point, thedisplay system may stop presentation of the virtual content based ondetection of a saccade. Examples of reducing resolution of virtualcontent are described in more detail in U.S. patent application Ser. No.15/927,808, which is hereby incorporated by reference in its entirety.

As will be described below, the display system may utilize one or moresensors, such as cameras configured to detect infrared, visible, and/orultraviolet light, to obtain sensor information, such as images, of theeyes of the user. For example, the sensors may be the camera assembly630 described above. While reference below is made to use of cameras, itshould be understood that additional sensors may be utilized to monitoreyes of the user. The display system may periodically obtain images ofone or both of the user's eyes, for example based on a refresh rate of30 Hz, 60 Hz, 120 Hz, and so on. Without being limited by theory, duringa saccade it is believed that both eyes of the user may follow a sametrajectory and perform the saccade at a same time. Thus, images of onlyone eye may advantageously be utilized. As described herein, optionallyimages of the other eye may be utilized for all other purposes (e.g.,gaze detection) or to improve detection of saccades.

Additionally, the display system may set an exposure time (e.g., shutterspeed) at a value such that motion of the eye may be captured. For aquick exposure time, the eye may be substantially frozen in the imagesuch that determining blur may be difficult. In contrast, for a longexposure time the image of an eye may have too much motion—thus reducinga likelihood of separating saccades from normal eye movement. In someembodiments, the exposure time may be set at 5 milliseconds, 10milliseconds, 15 milliseconds, and so on, such that the display systemmay detect motion blur that may correspond to a saccade. As describedherein, the exposure time may optionally be variable and dynamicallyadjusted by the display system.

The display system may optionally analyze captured images to identify aspecific portion of the eye, such as a pupil. For example, the displaysystem may utilize machine learning techniques to identify the pupil.Example machine learning techniques may include neural networks trainedto label portions of images that represent pupils. An example label mayinclude an outline presented on an image that corresponds to an outlineof a pupil. As another example, the display system may utilize edgedetection techniques (e.g., Canny edge detector, and so on) to extractan outline of features of the eye. The display system may then analyzethe outlines to determine an outline corresponding to a pupil (e.g.,based on geometry information associated with pupils). Furthermore, thedisplay system may utilize a zoom (e.g., optical or digital zoom) tofocus on the user's pupils. For example, the display system may identifya location of the user's pupils, and increase the zoom until imagesobtained of the eyes are substantially of the pupil. Optionally, thedisplay system may filter (e.g., remove) particular portions of theimage. For example, the display system may remove eyelashes, skin, andso on, from the images prior to further analysis. In this way, computervision and/or machine learning techniques employed thereafter may beimproved through removal of extraneous information.

To determine whether the user is performing a saccade, the displaysystem may determine measures or metrics associated with motion blur asrepresented in obtained images. For example, the display system mayidentify a saccade based on a single image. In other examples, thedisplay system may compare successive images. The display system maycompare these determined measures to one or more thresholds anddetermine whether the user is performing a saccade. For example, thedisplay system may determine a likelihood or confidence score associatedwith the user performing a saccade.

In some embodiments, the display system may determine a direction andvelocity of motion based on the determined motion blur. As will bedescribed below, the direction and velocity may be used to inform apredictive component for eye tracking, such as to estimate how long thesaccade might last for and where the saccade might end. Without beingconstrained by theory, the display system may utilize informationdescribing relationships between saccade velocity and amplitude (e.g.,angular distance). For example, a linear relationship may be knownbetween saccade velocity and amplitude. Thus, a single image capture(e.g., one image frame or sampling time) may be indicative of both theposition (e.g., current information) and the velocity (e.g., utilize topredict future information, such as where the saccade may end) of theusers eye.

As will be described below, the display system may utilize differentschemes to determine measures associated with motion blur. For example,the display system may analyze an image in the frequency domain, andidentify blur or rapid movement based on the frequency spectrum. Asanother example, the display system may perform convolutions of imagekernels with the image. These image kernels may be linearly convolved inthe time domain, or multiplied in the frequency domain, and resultscompared to thresholds. As another example, the display system maydecompose the image into eigen-images, for example via Singular ValueDecomposition. In this example, the decomposition may generateeigen-images, with the eigen-images presenting different scale spaceanalyses of the image. For example, a first few most significanteigen-images (e.g., larger singular values) may represent large scalefeatures of the image, while other eigen-images may represent finerdetail. Since blurred portions of an image may preserve large scalefeatures, while discarding finer detail, these eigen-images may beutilized to determine measures of blur.

Additional techniques to determine blur may be utilized, such astechniques based on geometrical information associated with a user'seyes. In this example, the display system may determine measures ofelongation associated with a user's pupil. For example, since theexposure time may be set such that motion is obtained, the user's pupilmay appear deformed in the image. The display system may determineinformation indicating a deformation of the pupil, and utilize theinformation to determine whether a saccade occurred. The deformation maybe correlated with an angular distance covered by the saccade, andtherefore for saccades of short duration (e.g., angular distance), thedeformation may be small (e.g., minimal). Thus, the display system mayoptionally utilize this deformation information in addition to othertechniques described herein (e.g., frequency spectrum techniques,kernel-based techniques, and so on).

As another example, the display system may monitor for changes in aquantity of identifiable eye features of the user between successiveimages. For example, eye features may comprise particular blood cells,iris pattern (e.g., folds, ridges), and so on. As mentioned above,example descriptions of eye features and identifying eye features isdescribed in more detail in U.S. Publication No. 2017/0109580, which isincorporated by reference herein in its entirety for all purposes. Inthis example, the display system may determine an occurrence of asaccade if in a particular image the display system may identify lessthan a threshold number of eye features. As an example, the displaysystem may be able to normally identify a particular number of eyefeatures, such as 20, 25, 30. During a saccade, these eye features mayappear blurry and thus may not be identified. Therefore, an image forwhich 10, 15, and so on, eye features may be identified, may beclassified as representing a saccade. While the display system maydetermine occurrence of a saccade based on a number of identified eyefeatures being less than a threshold, the display system may alsodetermine a saccade based on a difference in eye features identifiedbetween successive images being greater than a threshold. For example,the display system may identify a particular reduction in the number ofidentified eye features as corresponding to a saccade. In this way, thedisplay system may utilize the effects of motion blur (e.g., degradationof fine detail in images) to determine occurrence of saccades. In someembodiments, the display system may compare eye features of a useridentified in an image to known eye features of the user, such as thoserepresented in an iris code or a biometric authentication templatestored in association with the user. In some of these embodiments, thedisplay system may generate a confidence score indicating a level ofconfidence that an eye shown in an image matches a known eye of a userbased on such an eye feature comparison, and may determine occurrence ofa saccade based on the confidence score being less than a threshold. Insome examples, the display system may determine occurrence of a saccadebased on a relative change in confidence score being greater than athreshold. It follows that, in such embodiments, the set of known eyefeatures of the user may be selected through one or more biometricauthentication processes or otherwise ahead of time before saccadedetection processes are carried out. Example descriptions of eyefeatures, iris codes, and the generation of confidence scores inassociation therewith is described in more detail in U.S. PublicationNo. 2017/0206412, which is incorporated by reference herein in itsentirety for all purposes. Furthermore, additional information regardingblur metrics and the effect that motion blur may have on eye featuredetection processes can also be found in U.S. Publication No.2017/0206412.

In some examples, the display system may identify a contour of the pupilor iris using one or more machine learning and/or edge detectiontechniques. In a manner similar to that which has been described abovewith reference to eye feature detection, in some embodiments, thedisplay system may generate a confidence score indicating a level ofconfidence that one or more edges (e.g., along the border of a pupil,along the border of an iris, etc.) have been correctly identified, andmay determine occurrence of a saccade based on the confidence scorebeing less than a threshold. In some examples, the display system maydetermine occurrence of a saccade based on a relative change inconfidence score being greater than a threshold. The various thresholdsdescribed herein may be fixed values or may be variable values that aredynamically adjusted based on one or more parameters (e.g., userpreferences, user identity, preceding images, etc.).

In some embodiments, the display system may output light, such asinfrared light, and may detect reflections (e.g., corneal reflections or‘glints’) on the user's eyes in obtained images. A shape of these glintsmay be known to the display system, such that the display system maydetermine measures of deformity or blur of the glints. If the deformityor blur exceeds one or more thresholds, the display system may determinean occurrence of a saccade. Examples of infrared light images areillustrated in FIGS. 10A-10B. Additionally, the display system mayoutput infrared light as a band or strip of light. The band or strip oflight may encompass the user's iris and sclera (e.g., the limbicboundary). For example, FIGS. 15A-15B illustrate images having a band orstrip of light on a user's eye. In some embodiments, the angle at whichsuch light is projected onto the user's eye may be substantially uniformacross the band or strip. Although primarily described herein with bandor strip geometries, it is to be understood that any of a variety ofillumination geometries (e.g., an ellipsoid, a square, etc.) andpatterns (e.g., an array or row of distinct beams, a series of bands orstrips, etc.) may be projected over at least a portion of the limbicboundary of the user's eye. The display system may analyze the limbicboundary for changes (e.g., from a prior image), or to determinemeasures of motion blur of the limbic boundary. For example, the limbicboundary may deform and/or exhibit motion blur as the user moves his/hereyes (e.g., the user performs a saccade). It will be appreciated thatthe various techniques described herein for determining whether saccadeis being performed may be applied separately or in conjunction with oneor more other saccade detection techniques described herein. Utilizingmultiple techniques may advantageously increase the accuracy of thedetermination.

FIG. 10A illustrates example images 1002, 1004, of a user's eye asobtained by a display system (e.g., the display system 60) according toa first example scheme for detecting a saccade. As described herein, thedisplay system may utilize a camera to periodically obtain images of auser's eye. FIG. 10A illustrates a first image 1002 of the user's eye,for example obtained at a particular time associated with a refresh rateset by the display system (e.g., 30 Hz, 60 Hz, and so on as describedabove). The first image 1002 includes the user's eye 1006 along withinfrared glints 1008 reflected from infrared light output by the displaysystem on the eye 1006. As will be described in more detail below, thedisplay system may analyze this first image 1002 to determine whether asaccade is being performed. FIG. 10A further illustrates a second image1004 of the user's eye 1006. The display device may obtain this secondimage 1004 after a threshold amount of time (e.g., 0.333 seconds, 0.0166seconds, and so on). Similar to the first image 1002, infrared glints1008 are illustrated in the second image 1004.

In the first example scheme, to determine an occurrence of a saccade adisplay system may compare successive images of the user's eyes 1006and/or glints 1008. For example, the display system may generate adifference image 1010 representing a difference between the images 1004,1002. As illustrated, the difference image 1010 highlights distinctionsbetween the images 1004, 1002, with the distinctions between aparticular color (e.g., white). These distinctions may thereforerepresent movement between the images 1004, 1002. For example, theuser's eyelashes 1012 are illustrated as having moved. Similarly,movement in the glints 1014 is illustrated. The display system maydetermine degrees of movement, for example in the glints 1014, and basedon the movement determine whether a saccade occurred.

In this example scheme, since the difference image 1010 illustratessmall movements, the display system may determine that no saccade isbeing performed. For example, the display system may include thresholdsfor the lateral displacement of features in the difference image 1010.If the lateral displacement is less than the threshold, then the displaysystem may be configured to determine that no saccade is being performedby the eyes of the user. Instead, the display system may determine thata saccade is being performed in a third obtained image 1020 illustratedin FIG. 10B.

FIG. 10B illustrates additional example images 1004, 1020, of the user'seye. As illustrated, the third obtained image 1020 includes the user'seye 1006 and glints 1008. In this third image 1020, the glints 1008appear moved with respect to their positions in second image 1004. Thedisplay system may similarly generate a difference image 1022 betweenthe second image 1004 and third image 1020. This difference image 1022more strongly illustrates movement of eye features between the thirdimage 1020 and second image 1004, as compared to the difference image1010 illustrated in FIG. 10A.

Thus, based on difference image 1020, the display system may determinean occurrence of a saccade, e.g., because the movement of the glints1008 have been determined to move more than a predetermined threshold.As described above, the display system may therefore perform actionssuch as performing a depth plane switch. However, this example schememay result in detection of a saccade too late, such that the user maycomplete the saccade during performance of the actions. Thus, and as anexample, the user may be able to perceive the depth plane switch.However, based on the techniques described herein, for example at leastin FIG. 11A, the display system may advantageously be able to determineperformance of a saccade based on the second image 1004 (e.g., one framesooner), as early signs of saccadic movement may be expressed in thesecond image 1004. Thus, the display system may perform a depth planeswitch thereafter, such that as the saccade is being performed (e.g.,represented in the third image 1020) the depth plane switch will not benoticeable. To determine the occurrence of a saccade in the second image1004, the display system may determine measures associated with blur asdescribed herein.

FIG. 11A illustrates an example process 1100 for determining occurrenceof an event during which a user has reduced visual perceptibility. Forconvenience, the process 1100 may be described as being performed by adisplay system (e.g., the display system 60, which may includeprocessing hardware and software, and optionally may provide informationto an outside system of one or more computers or other processingdevice, for instance to offload processing to the outside system, andreceive information from the outside system).

At block 1102, the display system obtains an image of a user's eye. Asdescribed above, the display system may periodically obtain images foranalysis. For example, the display system may trigger a camera (e.g.,camera assembly 630) to capture the image and then obtain imageinformation from the camera. The display system may set an exposure timefor the image such that motion blur may be evident in the obtained image(e.g., 10 ms, 15 ms, and so on). Optionally, the frame rate at whichimages are obtained (e.g., 30 Hz, 60 Hz, and so on as described above)may remain static, or constant, while the exposure time may optionallybe variable (e.g., dynamically adjustable). Optionally, the frame ratemay be dynamically adjusted (e.g., based on operating parameters). For acamera having a shutter (e.g., a digital single lens reflex camera,particular mirrorless cameras, and so on), the camera may expose adigital sensor for the exposure time prior to closing the shutting. Fora camera that is shutterless (e.g., a mobile camera), the camera mayutilize light exposed on a digital sensor for the exposure time.

In some embodiments, the display system may dynamically adjust theexposure time. For example, an exposure time may be selected by thedisplay system (e.g., from a range of exposure times) based on a type ofinformation being determined. In the example of FIG. 11A, the displaysystem is determining occurrence of a saccade. In other exampleshowever, the display system may obtain an image to determine a gazeassociated with the user's eye. For example, the display system mayutilize a geometry of the user's eye to determine a vector extendingfrom the user's fovea. The display system may therefore select a shorterexposure time to reduce existence of motion blur. Additionally, thedisplay system may perform a biometric authentication process based onan image of the user's eye. For example, the display system may compareknown eye features of the user's user to eye features identified in theimage. Thus, the display system may similarly select a shorter exposuretime to reduce existence of motion blur.

When dynamically adjusting the exposure time, the display system mayalternate between a long exposure time mode and a short exposure timemode. In the long exposure mode, the display system may obtain imagesusing a first exposure time (e.g., 2 ms or more, 5 ms or more, 10 ms ormore, or 20 ms or more; including 2 ms, 5 ms, 10 ms, or 20 ms), and inthe short exposure mode, the display system may obtain images using asecond exposure time (e.g., 1.2 ms or less, 1 ms or less, 0.7 ms orless, 0.2 ms or less, or 0.1 ms or less; including 1.2 ms, 1 ms, 0.7 ms,0.2 ms, or 0.1 ms) that is shorter in duration than the first exposuretime. For example, the display system may obtain an image at the firstexposure time to determine whether the user is performing a saccade, andthen subsequently obtain an image at the second exposure time.Additionally, particular conditions of the display system or user mayinform whether images are to be obtained at the first or second exposuretime. For example, the display system may switch from the short exposuremode to the long exposure mode in response to determining that anaccommodation vergence mismatch exceeds one or more threshold values. Inthis way, the display system may begin to obtain images using the firstexposure time (e.g., to determine whether the user is performing asaccade) in anticipation of a depth plane switch. In some embodiments,the first exposure time may be greater than 1 ms (e.g., 2 ms or more, 5ms or more, 10 ms or more, or 20 ms or more; including 2 ms, 5 ms, 10ms, or 20 ms) and/or the second exposure time may be less than or equalto 1 ms (e.g., 1 ms or less, 0.7 ms or less, 0.2 ms or less, or 0.1 msor less; including 1 ms, 0.7 ms, 0.2 ms, or 0.1 ms). In some examples,the first exposure time may be greater than 1.2 ms (e.g., 2 ms or more,5 ms or more, 10 ms or more, or 20 ms or more; including 2 ms, 5 ms, 10ms, or 20 ms) and/or the second exposure time may be less than or equalto 1.2 ms (e.g., 1.2 ms or less, 1 ms or less, 0.7 ms or less, 0.2 ms orless, or 0.1 ms or less; including 1.2 ms, 1 ms, 0.7 ms, 0.2 ms, or 0.1ms).

In some embodiments, the display system may dynamically adjust one ormore of the exposure times. For example, the display system may increaseor decrease the first exposure time used for saccade detection. In thisexample, the display system may determine that measures associated withmotion blur are too high or too low. For example, the measures may notbe accurately detecting, or over detecting, saccades due to the exposuretime. For example, the display system may be configured to performsaccade detection using both motion blur detection and comparisonsbetween successively-captured image frames. Assuming that thecomparisons between image frames provide more accurate determination ofthe occurrence of saccades, the results provided by comparing multipleimageries may be used as a reference and the motion blur detection maybe adjusted until a desired (e.g., high) level of agreement is reachedbetween the results of the two schemes for saccade detection. If theimage frame comparison indicates that saccades are being under detected,the display system may be configured to increase the exposure time.Conversely, if saccades are being falsely detected, then the exposuretime may be decreased. In some embodiments, the display system mayselect one or more of the exposure times from a predetermined range ofexposure times. For example, the lower bound of such a predeterminedrange of exposure times may be 0.1 ms or 0.2 ms, and the upper bound ofsuch a predetermined range of exposure times may be 15 ms or 25 ms. Insome embodiments, the display system may select one or more of theexposure times from a plurality of different exposure times. Forexample, the plurality of different exposure times may include exposuretimes that are greater than or equal to 0.1 ms or 0.2 ms and less thanor equal to 15 ms or 25 ms.

In some embodiments, when performing a biometric authentication process,or when determining a gaze of the user, the display system may alsoadjust the exposure time. For example, the display system maydynamically reduce the exposure time to reduce motion blur, or thedisplay system may increase the exposure time if the obtained images arenot properly exposed (e.g., if the images are too dark).

In some embodiments, the display system may utilize the same camera foreach image obtained of the user's eye. For example, the display systemmay comprise a camera pointing at a particular eye of the user. Asdescribed above, it may be understood that when a user performs asaccade, both eyes may move in a corresponding manner (e.g., at asimilar velocity and amplitude). Thus, the display system may utilizeimages of the same eye to reliably determine whether a saccade is beingperformed. Optionally, the display system may comprise cameras pointingat each eye of the user. In such embodiments, the display system mayoptionally utilize the same camera to obtain images of the same eye ormay select a camera to utilize. For example, the display system mayselect a camera that is not being currently utilized. The display systemmay obtain images of the user's eyes for purpose other than determiningthe occurrence of saccades. As an example, the display system mayperform gaze detection (e.g., the display system may determine athree-dimensional point at which the user is fixating), biometricauthentication (e.g., the display system may determine whether a user'seye matches with a known eye), and so on. In some embodiments, when thedisplay system provides a command that an image is to be taken, one ofthe cameras may be in use. Therefore, the display system may select acamera not in use to obtain the image to be used for saccade detection.

Optionally, the display system may trigger both cameras to obtain imagesat the same time. For example, each camera may obtain an image at arespective exposure time. In this way, the display system may obtain afirst image of a first eye to determine measures of motion blur, whileobtaining a second image of a second eye to determine other information(e.g., information to be used for gaze detection, authentication, and soon). Optionally, both images may be utilized to determine whether theuser is performing a saccade. For example, and as will be described inmore detail below, the display system may determine deformation offeatures (e.g., an eye of the user, an infrared glint, and so on)illustrated in the first image as compared to the same features asillustrated in the second image. Optionally, the display system maycause each camera to alternate between two exposure values, for exampleout of phase from each other. For example, a first camera may obtain animage at a first exposure value, and at the same time a second cameramay obtain an image at a second exposure value. Subsequently, the firstcamera may obtain an image at the second exposure value, and the secondcamera may obtain an image at the first exposure value. In someembodiments, one or more of the display system architectures and methodsof operation described above may be employed to detect excessivemovements of the display system and/or user's head (e.g., movementscorresponding to reduced visual acuity or perceptibility) and adjust oneor more operating parameters on the basis thereof, as described infurther detail below with reference to FIGS. 16-19.

At block 1104, the display system determines whether motion blur isrepresented in the image. The display system may employ one or moreschemes to determine whether motion blur is present. The schemes may beapplied to portions of the image, such as the pupil, iris, limbicboundary, and so on as described herein. An example of the displaysystem identifying a pupil in the obtained image is described below,with respect to FIG. 11B. As described herein, example schemes mayinclude frequency spectrum-based techniques, convolutional techniques,deep learning or other machine learning techniques, geometricaltechniques, techniques that rely on identification of eye features,techniques that utilize infrared reflections from the user's eye, and soon.

The display system may optionally utilize a particular scheme based onparticular detected information. For example, one or more schemes may bepreferable when the user is in a dark area. In this example, the displaysystem may prefer to utilize techniques that utilize infraredreflections. Thus, prior to obtaining the image (e.g., described inblock 1102), the display system may optionally determine a measure ofbrightness of the ambient environment. The display system may then causeinfrared lights to be turned on (e.g., infrared glints, an infrared bandor strip), and thus reflected from the user's eye. As another example,the display system may be configured to utilize frequency spectrum-basedtechniques or convolutional techniques in nominal or bright lightingenvironments.

Additionally, the display system may request (e.g., periodically)whether the user was able to perceive actions performed in response todetection of saccades. For example, the display system may presentinformation to the user (e.g., virtual content) requesting whether theuser perceived a switch between a first depth plane and a second depthplane (e.g., over a period of time). In some embodiments, the user mayutilize his/her hands, or an input device, to specify whether the userwas able to perceive an action. The display system may thereforeidentify motion blur determination schemes that are associated with theleast perception of depth plane switches, or reductions in renderingquality. User responses may be aggregated, for example, by an outsidesystem, and information determined from the aggregation may be providedto the display system. For example, responses from multitudes of usersmay be aggregated to determine a most effective scheme.

Optionally, the display system may perform multiple of the schemes toimprove determination of motion blur. For example, the display systemmay perform each scheme (e.g., in parallel) and then compare resultsgenerated by each scheme. The display system may optionally assign aweight to each measure, and combine the measures to obtain an average orweighted average measure. The weights may optionally be determined basedon user responses to perception of depth plane switches or reductions inrender quality (e.g., aggregated user responses). This average orweighted average measure may then be compared to a threshold todetermine whether the measured blur is greater than the threshold, andthus indicates occurrence of a saccade. Optionally, the display systemmay utilize a voting scheme, with each motion blur determination schemeassociated with a vote. The display system may determine occurrence of asaccade based on a threshold number of the schemes voting that thedetermined motion blur is greater than a respective threshold.Optionally, each vote may be augmented according to a confidenceassociated with the determined measure. For example, the display systemmay determine measures of confidence for each motion blur scheme. As anexample, the display system may determine whether the spectral frequencyof the image strongly indicates motion along a particular direction witha particular magnitude. If the frequency spectrum indicates motion alongdifferent directions, the display system may assign a lesser confidenceand thus a lesser weight to this vote.

At block 1106, the display system determines whether the user performeda saccade based on determined motion blur. As described above, thedisplay system may determine measures of motion blur based on one ormore schemes. These determined measures may be compared to respectivethresholds to ascertain whether the measures indicate motion blur at asufficiently high level to correspond to a saccade. Optionally, thedisplay system may determine a confidence score or likelihood associatedwith the determination. For example, measures closer to respectivethresholds may be associated with reduced confidence scores orlikelihoods. The confidence score may optionally be determined viamachine learning techniques. For example, the display system may providethe determined measures of motion blur (e.g., via one or more motionblur determination schemes), optionally along with other information, toa machine learning model to determine the confidence score. Otherinformation may include environmental characteristics, such as light,temperature, and so on. Additionally, the information may describe thevirtual content being presented to the user. For example, the virtualcontent may be text in which the user is jumping his/her eyes around in.All of these variables may be taken into account by the machine learningmodel to provide a confidence score, which then may be used to providedifferent weights to the determinations made by different schemes.

In some embodiments, a user may set information indicating whether theuser prefers more or less aggressive saccade determinations. Forexample, a more aggressive saccade determination may enable a lowerconfidence score to cause the display system to perform actions, such asdepth plane switching or rendering quality reductions. It will beappreciated different levels of aggressiveness for the saccadedetermination may be tied to, e.g., the sensitivity of the user to depthplane switching and/or rendering quality reductions.

While the description herein has described depth plane switching beingperformed based on detection of a saccade, the display system mayautomatically switch a depth plane without detection of a saccade. Forexample, upon identifying that a switching of depth planes is to occur,the display system may store information (e.g., a flag) indicating thatupon detection of a saccade by the user, the display system is toperform the switch to the selected depth plane. If subsequently thedisplay system does not detect a saccade, the display system mayautomatically switch presentation to a new depth plane based on athreshold amount of time being exceeded.

At block 1108, the display system performs one or more actions inresponse to a positive determination of an occurrence of a saccade. Asdescribed above, the display system may delay switching from a firstdepth plane to a second depth plane until occurrence of a saccade.Additionally, the display system may reduce a rendering quality (e.g.,resolution) of virtual content during occurrence of the saccade.

FIG. 11 B illustrates another example process 1110 for determiningoccurrence of a saccade. For convenience, the process 1110 may bedescribed as being performed by a display system (e.g., the displaysystem 60, which may include processing hardware and software, andoptionally may provide information to an outside system of one or morecomputers or other processing device, for instance to offload processingto the outside system, and receive information from the outside system).

At block 1112 the display system obtains an image of a user's eye, forexample as described in FIG. 11A above regarding block 1102.

At block 1114, in some embodiments the display system identifies a pupilin the obtained image (e.g., an image with an exposure time such thatmotion blur may be evident). As described above, the display system mayutilize machine learning techniques or computer vision techniques toidentify the pupil. For example, a trained neural network may beutilized to identify a contour of the pupil. As another example, edgedetection techniques may inform the contour of the pupil. Thus, thedisplay system may isolate a region of interest for determiningperformance of a saccade. As will be described below, the display systemmay determine a measure of deformation or elongation of the pupil in theimage. Additionally, the display system may convolve an image portionillustrating the pupil, or portions thereof, with one or more imagekernels to determine measures of motion blur.

At block 1116 the display system determines motion blur represented inthe image, for example as described in FIG. 11A regarding block 1104. Atblock 1118 the display system determines whether the image illustratesperformance of a saccade. Block 1118 may be similar to block 1106 ofFIG. 11A.

At block 1120, the display system may utilize the obtained image fordisparate purposes in some embodiments. For example, differentprocessing modules or routines may opportunistically utilize the image.These different processing modules or routines may utilize the image inparallel with the display system determining whether the imageillustrates performance of a saccade by the user's eyes. Additionally,block 1120 may be performed subsequent to obtaining the image withoutwaiting for subsequent blocks to be performed (e.g., measures of motionblur). In some embodiments, block 1120 may be performed subsequent toblock 1116 or 1118, for example particular processing modules orroutines may utilize the information determined in blocks 1116 and 1118(e.g., the modules or routines may utilize a determined magnitude anddirection of motion blur).

In some embodiments, the display system may be configured to determine,at least in part, a gaze of the user at block 1120. For example, thedisplay system may determine a centroid of a fovea in a captured image.The display system may then identify a vector extending from the foveaas representing a gaze of the user's eye. If the display system has animage of the other eye, the display system may similarly determine avector extending from the other eye. An intersection inthree-dimensional space corresponds to a point at which the user isfixating.

In some embodiments, the display system may be configured to predict eyemovement at block 1120. As will be described in more detail below, withrespect to FIG. 12A, the display system may determine measures of motionblur based on a frequency spectrum scheme. The display system maydetermine a magnitude of the motion blur optionally along with adirection associated with the blur based on, for example, a Fouriertransform of the image. Based on the magnitude and direction, thedisplay system may estimate a location at which the eye is expected tofocus (e.g., subsequent to a saccade).

As described above, the velocity of a saccade may be correlated (e.g.,linearly correlated) with the amplitude of the saccade. Thus, based onthe exposure time of the image the display system may estimate velocityaccording to a determined magnitude of motion blur. The display systemmay correlate the determined magnitude with amplitude (e.g., angulardistance) to determine an estimated eye movement caused by the saccade.In this way, the display system may estimate a location at which theuser will be fixating subsequent to the saccade. In some embodiments,the location may indicate a vector along with a final three-dimensionalfixation point.

At block 1122 the display systems performs one or more actions inresponse to determining performance of a saccade, for example asdescribed above in FIG. 11A for block 1108. In some examples, theoperations described with reference to block 1120 may be performed afterthose described with reference to block 1122, performed independentlyfrom those described with reference to block 1122, or omitted fromexample process 1110.

Optionally, the display system may obtain a subsequent image (e.g., atthe static or fixed frame rate as described above), and analyze thesubsequent image. Optionally, the display system may discard theobtained image described in block 1102 (e.g., based on the image beingblurry, out of focus, or having one or more quality thresholds be belowa threshold). The display system may then utilize the subsequent image.

FIG. 12A illustrates a process 1200 for another example scheme todetermine performance of a saccade. For convenience, the process 1200may be described as being performed by a display system (e.g., thedisplay system 60, which may include processing hardware and software,and optionally may provide information to an outside system of one ormore computers or other processing device, for instance to offloadprocessing to the outside system, and receive information from theoutside system).

At block 1202, the display system obtains an image of a user's eyes. Asdescribed above, with respect to FIGS. 11A-B, the display system mayobtain images of the user's eye or eyes periodically. Block 1202 may besimilar to block 1102 of FIG. 11A. The display system extracts frequencyinformation from the obtained image at block 1204, for example byperforming a Fourier transform or Fast Fourier Transform. At block 1206,the display system determines measures associated with motion blur ofthe user's eye. For example, the display system may determine amagnitude and direction associated with the motion blur which will bedescribed in more detail below with respect to FIG. 12B.

Reference will now be made to FIG. 12B, which illustrates an exampleblock diagram of a process 1210 to determine a magnitude and directionassociated with motion blur. At block 1212, the display system obtainsan image of the user's eye and at block 1214 the display systemoptionally identifies a pupil included in the image. At block 1216, thedisplay system performs a Fast Fourier Transform to obtain frequencycomponents of the image. For example, the display system may generate anadjusted image that includes values corresponding to the frequencycomponents of the obtained image. At block 1218 the display systemdetermines the log spectrum of the Fourier transformed image, and atblock 1220 the display system converts the log spectrum into binary.

To determine a direction associated with motion blur in the image, atblock 1222 the display system performs a Hough transform to provide anaccumulator array. It will be appreciated that the Hough transform maydivide the blur parameter space into accumulator cells. A given cell inthe accumulator may indicate a total number of curves passing throughit. To determine the direction, the display system identifies themaximum value in the accumulator, which corresponds to the direction ofthe motion blur (and thus the direction of eye movement).

To determine a magnitude associated with motion blur, at block 1224 thedisplay system obtains information identifying the determined direction,and rotates the binary spectrum (e.g., determined in block 1220) in adirection opposite to the blur direction. The display system may thencollapse the determined rotated information into one-dimension viataking an average along the columns of the two-dimensional rotatedinformation. Subsequently, the display system may perform an InverseFourier Transform on the one-dimensional information and identify afirst negative value in a real part. This identified first negativevalue corresponds to a length associated with the blur, and thus amagnitude.

With reference to FIG. 12A, at block 1208 the display system maydetermine whether the user performed a saccade based on the determinedmeasures. For example, if the determined magnitude is greater than athreshold the display system may identify occurrence of a saccade.

As described above with respect to FIGS. 11A-B, the display system mayopportunistically utilize the obtained image and determined measuresassociated with motion blur to estimate eye movement of the user. Forexample, the display system may monitor a user's gaze and determinethree-dimensional fixation points at which the user is fixating. If thedisplay system identifies occurrence of a saccade, it may represent thatthe user's eyes are to rapidly move until the user regains focus at aparticular three-dimensional location. As described in FIG. 12B, thedisplay system may determine parameters associated with motion blur of auser's eye (e.g., a user's pupil). For example, the display system maydetermine a magnitude and direction of the motion blur. In someembodiments, the display system may store and/or access one or morepredetermined models to map motion blur to magnitude and direction.Using these determined parameters, the display system may thereforeadjust a previously determined three-dimensional fixation point toidentify an estimated three-dimensional location at which the user willbe fixating upon completion of the saccade. As an example, the magnitudedetermined in block 1224 may be associated with a velocity of thesaccade. As described above, the velocity may be correlated with anamplitude of the saccade (e.g., linearly correlated), and thus thedisplay system may determine an estimated angular distance the user'seye will move during the saccade. The estimated angular distance may beapplied in a direction as determined in block 1222.

Advantageously, the display system may utilize an image of one eye ofthe user. For example, as described above during a saccade both eyeswill typically move together. Thus, the display system may adjust apreviously determined three-dimensional fixation point with parametersassociated with motion blur of a determined magnitude and direction ofone of the user's eyes. Since the other eye may be assumed to movesimilarly, the display system may determine the estimatedthree-dimensional location at which the user will be fixating after thesaccade.

FIG. 13 illustrates a process 1300 for another example scheme todetermine performance of a saccade. For convenience, the process 1300may be described as being performed by a display system (e.g., thedisplay system 60, which may include processing hardware and software,and optionally may provide information to an outside system of one ormore computers or other processing device, for instance to offloadprocessing to the outside system, and receive information from theoutside system).

At block 1302, the display system obtains an image of a user's eyes.Block 1302 may be similar to block 1102 of FIG. 11A. As described aboveregarding FIGS. 11A-B, the display system may obtain images of theuser's eye or eyes periodically. Additionally, the display system mayidentify a particular region of the user's eye, such as a pupil. Atblock 1304, the display system accesses information identifying one ormore image kernels to be convolved with the obtained image. For example,the kernels may comprise a laplacian kernel (e.g., a three by threeimage matrix), a Gabor filter (e.g., a Gaussian kernel functionmodulated by a sinusoidal plane wave), directional filter, and so on.Optionally, the display system may perform Singular Value Decompositionof the image to determine estimated motion blur, for example asdescribed above. Optionally the image kernels may be selected based onoperating or environmental conditions associated with the display system(e.g., a particular image kernel may be preferred at night time). Insome embodiments, the image kernels may be stored in local memory (e.g.,in a memory unit, containing e.g. volatile or non-volatile memory,associated with the local processing and data module 140). In someembodiments, the display system may perform processing for block 1304locally (e.g., on the local processing and data module 140).

At block 1306, the display system convolves the obtained image, or aportion thereof (e.g., the pupil), with one or more of the imagekernels. Based on the convolution, the display system may determineestimated motion blur represented in the image. For example, theconvolution may result in a measure, which may be a ratio or otherdimensionless value. At block 1308, the display system determinesoccurrence of a saccade based on the determined measure exceeding athreshold.

FIGS. 14A-14B illustrate another example scheme to determine performanceof a saccade based on reflected infrared light. FIG. 14A illustrates afirst image 1402 of an image of a user's eye 1404. For example, thefirst image 1402 illustrates a pupil 1406 of the eye 1402. Similar tothe description above in FIGS. 10A-10B, the display system may projectlight (e.g., infrared light) onto the user's pupil 1406. When imagingthe user's eye 1404, the infrared light may be reflected as one or moreglints 1408. For example, four infrared lights may be utilized. In thisexample, as the user's eye 1402 moves about particular glints may beimaged by the display system while other glints may not be included. Inthe example of FIG. 14A, two glints 1408 are illustrated. A shape ofthese glints may be known to the display system, such that the displaysystem may determine measures of deformation (e.g., elongation) whenimaged by the display system. Optionally, the shape may be determinedbased on capture of an image with a short exposure time (e.g., asdescribed above), such that the glints are imaged without motion blur.As described above, the images obtained of the user's eye 1402 may havean exposure time set such that motion blur is evident in the images.Thus, the display system may determine measures of motion blur of theuser's eye 1402 based on motion blur of the glints 1406.

FIG. 14B illustrates a second image 1410 of the user's eye 1402. Asillustrated, the glints 1408 are illustrated as being deformed and/orexhibiting motion blur. The display system may determine measuresassociated with the motion blur, for example based on the techniquesdescribed herein. As an example, the display system may isolate theglints 1408, and determine measures of motion blur of the isolatedglints 1408. The display system may determine a degree to which theglints 1408 appear deformed, for example determine elongation of theglints 1408. In some embodiments, the display system may leverage one ormore ellipse-fitting techniques to identify and/or evaluate elongatedfeatures (e.g., glints, a user's pupil, a user's iris, etc.) in capturedimages. Based on the deformation, the display system may estimate motionblur. Additionally, based on a direction of the elongation ordeformation the display system may estimate a direction of the motionblur.

FIGS. 15A-15B illustrate another example scheme to determine performanceof a saccade based on reflected light, e.g., infrared light. In theexample of FIGS. 15A-15B, a band or strip of light 1508 (e.g., infraredlight) is projected onto a user's eye 1504. FIG. 15A illustrates a firstimage 1502 of the user's eye 1504, with a limbic boundary 1506 of theeye 1504 (e.g., a sclera to an iris) illuminated by the band or strip oflight 1508. Similar to the description of FIGS. 14A-14B, the displaysystem may analyze the first image 1502 for significant temporal changesand/or the presence of motion blur based on the band or strip of light1508. Based on a shape of the user's eye 1504 (e.g., which include acorneal bulge), indicia of quick eye movements may be more readilyapparent in regions of images within which the limbic boundary 1506 isshown. For example, the user's eye 1504 may reflect portions of band orstrip of light 1508 incident the iris or corneal bulge of the user's eye1504 and portions of band or strip of light 1508 incident the sclera ofthe user's eye 1504 in different manners (e.g., at different angles). Assuch, the appearance of a portion of band or strip of light 1508 thatspans the limbic boundary 1506 may change relatively drastically as theuser's eye 1504 moves. In some instances, the effects of motion blur(e.g., as the user's eye 1504 engages in saccadic movement) may serve tofurther amplify such changes in appearance. FIG. 15B illustrates asecond image 1510 of the user's eye 1504. In this image 1510, the bandor strip of light 1508 is imaged as being deformed and/or exhibitingimage blur at the limbic boundary 1506. The display system may determinemeasures associated with blur, as described above, and/or may determinedeformation or elongation of the band or strip of light 1508.

Other Example Schemes to Switch Depth Planes

As described above, a display system (e.g., display system 60) maydetect saccades, and cause the presentation of virtual content to beadjusted from a first depth plane to a second depth plane upon detectionof the saccades, such that the adjustment, or depth-plane switching,preferably occurs during the saccade and is masked by the saccades. Forexample, the display system may determine saccades via motion blurestimation techniques (e.g., as described in FIGS. 11A-12B). As otherexamples, the display system may utilize image kernels (e.g., asdescribed in FIG. 13), glints reflected from a user's eye (e.g., asdescribed in FIGS. 14A-14B), or light reflected from the user's eyewhich spans particular physiological features (e.g., infrared lightspanning a limbic boundary as described in FIGS. 15A-15B). Since auser's visual acuity or perceptibility may be reduced during performanceof a saccade, the display system may thus advantageously perform a depthplane switch at a time when the perceptibility of the depth-plane switchis reduced.

It will be appreciated that a user may regularly or occasionally notperform saccades over a time scale that coincides with the time scaledesired for depth plane switching. When a user is not performingsaccades, however, the advantageous masking of depth plane switches bysaccades is not available. Optionally, in addition or as an alternativeto detecting saccades, the display system may monitor for occurrences ofblinks as a masking event. For example, U.S. Patent Pub. 2017/0276948(the '948 publication) describes detecting blinks and performing depthplane switches during the blink, and is incorporated by reference hereinin its entirely. As described in the '948 publication, a depth planeswitch that needs to be performed may be flagged (e.g., identified) bythe display system. Just as users may not perform saccades, users mayavoid blinking for extended durations in some instances. If the userdoes not blink or perform a saccade for a threshold amount of time(e.g., 10 seconds, 30 seconds, 60 seconds), the display system mayperform the flagged depth plane switch.

Additional techniques to cause the switching of depth planes may beemployed in addition to detection of saccades, detection of blinks, andswitching after the non-detection of blinks and/or saccades (e.g., asdescribed herein) for a threshold amount of time. As a first example,the display system may determine measures associated with its movement.Based on determined movement of the display, the display system mayperform a depth plane switch. As will be described, movement of thedisplay may be associated with reduced visual acuity or perceptibilityof a user. For example, if the user is moving their head rapidly (e.g.,at greater than a threshold velocity, angular velocity, and so on), thelikelihood of noticing a depth plane switch may decrease, since theuser's eyes are likely also moving.

As a second example, the display system may determine whether anaccommodation-vergence mismatch exceeds a threshold. Based on thethreshold being exceeded, the display system may perform a depth planeswitch. As described in FIGS. 4A-4D, an accommodation-vergence mismatchmay be caused by vergence cues associated with displayed virtual contentindicating a depth (distance away from a wearer) which is mismatchedfrom the accommodation cues associated with the depth plane on which thevirtual content is presented. For example, for situations in whichvirtual content is to be presented at depths that are outside of aparticular threshold depth range (e.g., diopter range) associated withacceptable accommodation-vergence mismatches for a selected depth plane,the display system may perform a depth plane switch. This depth planeswitch may improve the viewing comfort of the user by avoiding excessiveaccommodation-vergence mismatches.

As a third example, the display system may monitor a gaze of the user(to determine a three-dimensional point at which the user is fixating),and may perform a depth plane switch based on whether the user isfixating on virtual content. For example, the display system maydetermine a three-dimensional dista-ce between the user's fixation pointand a three-dimensional location at which virtual content is configuredto be presented. As an example, if the display system determines thatthe virtual content is being presented at greater than a thresholdangular distance from the user's fixation point, a depth plane switchmay be performed. In this example, the virtual content may be determinedto be presented outside of the user's fovea. Without being limited bytheory, for such content outside of the fovea, the user's visual acuityor perceptibility may be reduced and the depth plane switch may be lessnoticeable. As another example, if the display system determines thatthe virtual content is outside of a zone at which the user is fixating,the display system may perform the depth plane switch. In this example,the user's field of view may be separated into zones of contiguousthree-dimensional space. Thus, the display system may determine whetherthe user is fixating outside of a zone that includes virtual content. Asanother example, the display system may perform a depth plane switchbased on the distance (e.g., three-dimensional distance) between theuser's fixation point and virtual content exceeding a thresholddistance. Example descriptions of zones in three-dimensional space, aswell as systems and techniques for determining and evaluating fixationpoints and accommodation vergence mismatches relative to zones inthree-dimensional space and virtual content and taking action on thebasis thereof are described in more detail in U.S. application Ser. No.16/353,989, which is hereby incorporated by reference in its entiretyfor all purposes. In some embodiments, one or more the abovementionedsystems and techniques may be employed in or included as part of one ormore of systems and techniques described herein. Furthermore, in someembodiments, one or more of the zones described herein may correspond toone or more of the above-mentioned zones in three-dimensional space.

One or more of the above-described examples may thus be utilized by thedisplay system to determine a time at which to perform a depth planeswitch. These examples may be performed at the exclusion of othertechniques for determining the timing of depth plane switching, or oneor more of these examples may also be combined with the varioustechniques described herein. For example, the display system may monitorfor the performance of saccades and/or blinks in addition to any of theexamples described. In addition, in some embodiments, the display systemmay monitor the amount of time that has elapsed between a flagindicating that in depth plane switch is desired and the occurrence of asaccade or blink. Once the amount of time has passed the temporalthreshold for saccade and/or blink detection, and no saccade and/orblank has been detected, the display system may then proceed to switchdepth planes based upon other techniques described herein (e.g., depthplane switching may occur based upon detected movement, potentialaccommodation-vergence mismatch, or fixation away from virtual content).In some embodiments, the first example described above may be utilizedby the display system to determine a time at which to adjust one or moreoperating parameters thereof. For example, the first example describedabove may be utilized by the display system to determine a time at whichthe usage of computational resources and/or power may be reduced. Insome embodiments, such a reduction may occur for a duration of adetermined movement of the display. For example, the display system mayadjust one or more operating parameters, such as a virtual contentrendering setting (e.g., reduce a resolution at which virtual content isbeing rendered), a display setting (e.g., reduce a refresh rate of thepresented virtual content, a brightness, contrast ratio, color settings,and so on), and/or a power setting (e.g., clock speed of the CPU and/orGPU may be temporarily reduced, the display may be turned off, and soon). The first example described above may be performed at the exclusionof other techniques for determining the timing of the adjustment of oneor more operating parameters of the display system, or the first exampledescribed above may also be combined with one or more of the varioustechniques described herein for determining the timing of the adjustmentof one or more operating parameters of the display system.

As will be described, the display system may optionally prefer certainschemes to cause a depth plane switch than other schemes. As an example,the display system may access information indicating a precedencenetwork. The precedence network may inform when to cause a depth planeswitch. For example, the display system may prefer switching upondetection of a saccade over switching upon determining that anaccommodation-vergence mismatch exceeds a threshold.

FIG. 16 illustrates an example process 1600 for performing depth planeswitching based on detected movement. For convenience, the process 1600may be described as being performed by a display system (e.g., thedisplay system 60, which may include processing hardware and software,and optionally may provide information to an outside system of one ormore computers or other processing device, for instance to offloadprocessing to the outside system, and receive information from theoutside system).

At block 1602, the display system obtains information indicative ofmovement. As described above, for example with respect to the FIG. 9D,the display system may include inertial measurement units,accelerometers, compasses, GPS units, radio devices, gyros, and/or othersensors. The display system may obtain information from these sensors toobtain movement information. As an example, the display system mayperiodically monitor information being received from these sensors.Optionally, the display system may be interrupted by information beingprovided by the sensors. For example, if the information indicatesmovement greater than a threshold, or greater than a threshold velocity,the sensors may trigger an interrupt for the display system to process adepth plane switch.

The display system may also utilize cameras to obtain movementinformation. For example, as described above, the display system mayinclude environmental sensors (e.g., sensors 112).

These sensors may include outward facing cameras that can obtain imagesor video of a real-world environment in which a user is located. Thedisplay system may, as described above, be configured to detect light,objects, stimuli, people, animals, locations, or other aspects of theworld around the user. Additionally, the display system may detectcertain fixed features included in images of the world surrounding theuser. For example, the display system may monitor for corners, edges,contours and/or other fixed features which can be tracked. Thisinformation may be utilized to determine estimates associated withmovement. For example, if certain fixed features moved a thresholdthree-dimensional distance in a certain period of time, the displaysystem may estimate velocity of movement along with a vector describingthe movement. Example descriptions of features of real-worldenvironments, as well as systems and techniques for identifying andmapping real-world environments, generating and accessing models ofreal-world environments, recognizing or otherwise identifying objectsand features of real-world environments are described in more detail inU.S. Publication No. 2015/0302652, U.S. Publication No. 2017/0091996,U.S. Publication No. 2017/0301133, U.S. Publication No. 2018/0045963,and U.S. Publication No. 2018/0268220, all of which are incorporated byreference herein in their entirety for all purposes. In someembodiments, one or more the abovementioned systems and techniques maybe employed in or included as part of one or more of systems andtechniques described herein. Furthermore, in some embodiments, one ormore of the fixed features described herein may correspond to one ormore of the abovementioned features of real-world environment.

At block 1604, the display system determines measures associated withmovement. As described above, the display system may utilize, asexamples, certain sensors and/or cameras to detect movement. Forexample, the display system may obtain information from an accelerometerand/or inertial measurement unit to estimate movement of the displaysystem. It should be appreciated that an inertial measurement unit maymeasure the display system's specific force, angular rate, andoptionally magnetic field surrounding the display system. The displaysystem may utilize one or more of accelerometers, gyroscopes, andmagnetometers, and multiple sensors may be utilized to increase theconfidence or accuracy of the movement detection. Thus, the displaysystem may determine estimates as to an extent of the movement (e.g., athree-dimensional distance traveled) and velocity and/or acceleration ofthe movement.

Additionally, as noted herein, the display system may utilize cameras todetermine estimates of movement, velocity, and/or acceleration. Forexample, the display system may track certain fixed fixtures in areal-world environment surrounding the user. As another example, thedisplay system may utilize one or more of the techniques described abovein FIGS. 11A-15. As an example, the display system may determine motionblur in one or more images obtained via outward facing cameras. Asdescribed in FIG. 12A, the display system may utilize frequencyextraction to determine estimates of motion blur. Other schemes that maybe employed in the display system for determining motion blur includeconvolutional techniques, determining elongation of one or moreidentifiable objects and features of real-world environments, deeplearning techniques (e.g., neural network-based image analysis), and soon. While these estimates may not be utilized to determine the displaysystem's location in three-dimensional space (e.g., as compared to aninertial measurement unit), the estimates may inform an extent to whichthe system moved. Additionally, the estimates may inform a velocityand/or acceleration of the movement. As another example, the displaysystem may track fixed features in a real-world environment surroundingthe user over time to determine whether the display system and/or user'shead is engaged in a threshold measure or more of movement in much thesame way that the display system may track eye features of one or bothof the user's eyes to determine whether one or both of the user's eyesare engaged in a threshold measure or more of movement (e.g., saccadicmovement). More specifically, in some embodiments, the display systemmay monitor for changes in a quantity of identifiable fixed features inthe real-world environment (e.g., features that the display system iscapable of identifying with a degree of confidence that exceeds one ormore thresholds) between successive images captured by one or moreoutward facing cameras to determine whether the display system and/oruser's head is engaged in a threshold measure or more of movement. Insuch embodiments, the display system may compare fixed featuresidentified in one image of the real-world environment to fixed featuresidentified in a previously-captured image of the real-world environment.For example, the display system may identify a particular reduction inthe number of identified fixed features from one image to another ascorresponding to a threshold measure of movement. In this way, thedisplay system may utilize the effects of motion blur (e.g., degradationof fine detail in images) to detect excessive movements of the displaysystem and/or user's head (e.g., movements corresponding to reducedvisual acuity or perceptibility).

In some embodiments, the display system may compare fixed featuresidentified in an image of the real-world environment to known featuresof the real-world environment, such as those represented in one or moremodels stored in association with the user and/or the geographiclocation of the real-world environment. In some of these embodiments,the display system may generate a confidence score indicating a level ofconfidence that a fixed feature shown in an image matches a knownfeature of the real-world environment based on such a featurecomparison, and may detect occurrences of excessive movements of thedisplay system and/or user's head based on the confidence score beingless than a threshold.

In some examples, the display system may identify a fixed feature (e.g.,corner, edge, contour, etc.) of a real-world environment using one ormore machine learning and/or edge detection techniques. In a mannersimilar to that which has been described above with reference to eyefeature detection, in some embodiments, the display system may generatea confidence score indicating a level of confidence that one or morefixed features have been correctly identified, and may determineoccurrences of excessive movements of the display system and/or user'shead based on the confidence score being less than a threshold. In someexamples, the display system may determine occurrences of excessivemovements of the display system and/or user's head based on a relativechange in confidence score being greater than a threshold. In someexamples, the display system may determine such occurrences based on arelative change in confidence score being greater than a threshold. Thevarious thresholds described herein may be fixed values or may bevariable values that are dynamically adjusted based on one or moreparameters (e.g., user preferences, user identity, preceding images,etc.).

As mentioned above, in some embodiments, one or more of the displaysystem architectures and methods of operation described above withreference to FIG. 11 may be employed to detect excessive movements ofthe display system and/or user's head (e.g., movements corresponding toreduced visual acuity or perceptibility) and adjust one or moreoperating parameters on the basis thereof. As an example, in suchembodiments, one or more outward facing cameras of the display systemmay be configured to capture images of a real-world environment using astatic frame rate and variable exposure time, and the display system maybe configured to control such one or more outward facing cameras todynamically adjust the exposure time based on one or more conditions ofthe display system or the user. In some examples, such one or moreconditions of the display system or the user may include one or moreprocesses to be performed by the display system (e.g., biometricauthentication, processes to detect reduced visual acuity orperceptibility, etc.), a determined fixation point of the user, adetermined accommodation vergence mismatch, and the like.

As another example, in at least some of these embodiments, the displaysystem may be configured to switch or alternate between a short exposuretime mode and a long exposure time mode, as described in further detailabove with reference to FIG. 11. In some such embodiments, the displaysystem may be configured to control at least two outward facing camerasto dynamically adjust the exposure time based on one or more conditionsof the display system or the user. Other display system architecturesand methods of operation described herein may be employed to detectexcessive movements of the display system and/or user's head.

At block 1606, the display system causes a depth plane switch. Thedisplay system may determine that the movement exceeds one or morethresholds for movement and performs a depth plane switch. For example,the display system may flag, or otherwise identify that depth planeswitch is due to be performed. The display system may then perform theswitch based on determining that movement of the display system isgreater than certain thresholds. The thresholds may optionally beuser-selectable, or may be learned over time for a specific user orbased on aggregation of information obtained from users. For example,the display system may request information from the user indicatingwhether the user could see the depth plane switch. As discussed herein,it will be appreciated that switching depth planes involves changing thewavefront divergence of light used to form an image corresponding tovirtual content.

As an example, a threshold may indicate that if the movement is beingperformed at greater than a threshold velocity, then the depth planeswitch may be performed. As another example, a threshold may indicatethat if the user moved greater than a threshold distance, then the depthplane switch may be performed. As another example, a threshold mayindicate that if the movement is being performed at greater than athreshold acceleration, the depth plane switch may be performed.Optionally, one or more of these thresholds may be combined or berequired to be satisfied. For example, the depth plane switch may beperformed if the movement indicates the display system is moving atgreater than at threshold velocity and also increasing in velocity atgreater than a threshold acceleration. The thresholds may furtherrequire a time element. For example, a depth plane switch may beperformed if the display system is moving greater than a thresholdvelocity for greater than a threshold period of time. Similarly, thedisplay system may perform the depth plane switch if the display systemis moving at greater than a threshold acceleration for greater than athreshold period of time.

FIG. 17 illustrates an example process 1700 for performing depth planeswitching based on an accommodation-vergence mismatch. For convenience,the process 1700 may be described as being performed by a display system(e.g., the display system 60, which may include processing hardware andsoftware, and optionally may provide information to an outside system ofone or more computers or other processing device, for instance tooffload processing to the outside system, and receive information fromthe outside system).

At block 1702, the display system presents virtual content at aparticular depth plane. As described herein, the display system maypresent virtual content at various depths utilizing different depthplanes (e.g., by outputting light from different waveguides that providedifferent amounts of wavefront divergence).

At block 1704, the display system adjusts virtual content in depth; thatis, the display system adjusts the perceived depth or distance of thevirtual content from the user. For example, vergence cues may beadjusted to cause the user to perceive the virtual content at theadjusted depth.

While the vergence cues may be adjusted, the accommodation cues mayremain the same until the display system switches depth planes from onedepth plane (corresponding to one set of accommodation cues) to anotherdepth plane (corresponding to another set of accommodation cues).Consequently, depending upon the distance from the user that virtualcontent is placed, the changes in vergence cues may cause mismatcheswith accommodation cues.

At block 1706, the display system determines the degree of theaccommodation-vergence mismatch. As described with reference to FIGS.4A-4D, the display system may monitor a mismatch between the vergencecues and accommodation cues for virtual content being fixated on by theuser. The mismatch may be measured in diopters, as also describedherein.

At block 1708, the display system performs a depth plane switch upondetermining that the accommodation-vergence mismatch threshold has beenexceeded. It will be appreciated that the human visual system maytolerate limited levels of accommodation-vergence mismatches and that,above certain thresholds (e.g., 0.25 diopter, 0.33 diopter, or 0.5diopter), the user may experience significant discomfort. The displaysystem may determine whether the accommodation-vergence mismatch forparticular virtual content has exceeded a predefinedaccommodation-vergence mismatch threshold. If the threshold is exceeded,the display system performs the depth plane switch. The switch may beperformed irrespective of whether a masking event is occurring. Whilethe user may, as an example, be able to perceive the switch, the switchmay be advantageous to the performance of the display system. Forexample, without being limited by theory, perception of depth planeswitching may negatively impact the realism or immersion of the user ina viewing experience, but high levels of accommodation-vergencemismatches is more undesirable since it made cause sufficient discomfortto stop the user from using the display system.

In some embodiments, the accommodation-vergence mismatch thresholds thattrigger depth plane switching may be set at constant values for allusers. In some other embodiments, the thresholds may vary between users.For example, certain users may be more able to tolerate higher mismatchlevels than others. Thus, the threshold may be adjusted for these users(e.g., the users may adjust the threshold in some embodiments). In someembodiments, the thresholds may vary depending on the depth of virtualcontent from the user. For example, for distances closer to the user,the thresholds may be smaller as these mismatches may be more detectableto the user.

FIG. 18 illustrates an example process 1800 for performing depth planeswitching based on a determined fixation point of a user. Forconvenience, the process 1800 may be described as being performed by adisplay system (e.g., the display system 60, which may includeprocessing hardware and software, and optionally may provide informationto an outside system of one or more computers or other processingdevice, for instance to offload processing to the outside system, andreceive information from the outside system).

At block 1802, the display system causes presentation of virtualcontent. Similar to blocks 1602, 1702, the display system may presentvirtual content to be perceived at different three-dimensional locationsfrom the user.

At block 1804, the display system determines a fixation point of theuser. As described above, the display system may be configured todetermine, at least in part, a gaze of the user. For example, thedisplay system may determine a centroid of a fovea in a captured image.The display system may then identify a vector extending from the foveaand through the pupil as representing a gaze of the user's eye. If thedisplay system has an image of the other eye, the display system maysimilarly determine a vector extending from the other eye. Anintersection in three-dimensional space may thus correspond to a pointat which the user is fixating.

At block 1806, the display system determines information indicative oflocation differences between the fixation point and virtual content. Thedisplay system can access information identifying three-dimensionallocations of virtual content. The display system may then compare theselocations to the determined fixation point. As an example, the displaysystem may determine an angular difference (e.g., with respect to theuser's eyes) between the fixation point and virtual content. As anotherexample, the display system may determine whether the virtual contentfalls on a fovea of the user. As another example, the display system maydetermine a three-dimensional distance (e.g., based on a vector norm)between the virtual content and fixation point. Thus, these examples maythus inform whether the user is actually looking at, or fixating on, thevirtual content.

Additionally, the display system may utilize images of the user's eyesto determine a likelihood that the user is fixating on some real-worldfeature. For example, if the user is focusing within a certain volume ofthree-dimensional space, and is not moving, or is barely moving, his/hereyes outside of the volume, then the display system may increase thelikelihood. Thus, the user may be determined to not be fixating on, orbe much aware of, the virtual content may be, e.g., moving. Thislikelihood may inform whether to cause the depth plane switch. Forexample, if the user is determined to be likely focusing on a stationaryreal feature, then the display system may perform a depth plane switchfor virtual content, which may be at other locations.

As described above, the user's field of view may optionally be separatedinto different zones of space. For example, the field of view may beseparated into three-dimensional polygons. Example polygons may include,cubes, hyperrectanges, toruses, spheres, and so on. The display systemmay determine a polygon in which the user is fixating, and a polygon inwhich the virtual content is presented.

At block 1808, the display system performs a depth plane switch. Thedisplay system may compare the determined information in block 1806 toone or more thresholds or rules and determine whether to perform a depthplane switch. As an example, the display system may cause a depth planeswitch if an angular distance between the virtual content and fixationpoint exceeds a threshold. As another example, the display system maycause a depth plane switch if the virtual content is located in adifferent zone than the fixation point, or at greater than a thresholdnumber of zones from the fixation point. Optionally, the thresholdnumber may be required to be along a certain direction, such that thevirtual content falls outside of the user's fovea.

FIG. 19 illustrates an example block diagram 1900 of a display systemconfigured to determine a time at which to perform a depth plane switchusing the various techniques disclosed herein. For example, the displaysystem may be the display system 60 described herein.

As illustrated in FIG. 19, to determine whether to cause a depth planeswitch, the display system may:

-   -   perform saccade detection 1902 (e.g., as discussed with        reference to FIGS. 10A-15B),    -   perform blink detection 1903,    -   determine the presence of excessive accommodation-vergence        mismatches 1904 (e.g., as discussed with reference to FIG. 17),    -   detect movement 1906 that may mask a depth plane switch (e.g.,        as discussed with reference to FIG. 16),    -   determine the gaze of a user 1908 and whether the user is        fixating on virtual content (e.g., as discussed with reference        to FIG. 18), and    -   determine whether a threshold amount of time as elapsed since        the display system indicating a depth plane switch should occur        1910 (e.g., a time since the system 60 flagged the switch).

The display system 60, upon identifying that a switch is flagged tooccur, may thus monitor for these example schemes to cause a depth planeswitch 1912. As described above, the display system may prefer certainexample schemes over others. For example, the display system may prefera saccade 1902 over the remaining schemes. As another example, thedisplay system may prefer a blink over the other schemes 1902-1910.Alternatively, the display system may be configured detect and act uponother masking events before detecting saccades and/or blinks. As anotherexample, the display system may traverse a precedence network. Theprecedence network may inform which schemes are favored over otherschemes, preferentially performing certain schemes before other schemes(e.g., performing saccade or blink detection for a set amount of timebefore performing the other schemes).

Optionally, the display system may learn preferences of the user. Forexample, the display system may request information from the userregarding whether the user could perceive a depth plane switch.Optionally, the display system may utilize machine learning models tolearn the user's behavior. The display system may further receiveaggregated information from multitudes of users to learn preferences.For example, the precedence network may be updated based on thelearning. The display system may periodically connect to an outsidesystem for updates to the precedence network. In this way, the displaysystem may cause depth plane switches 1912 while increasing likelihoodthat the user is not able to perceive the switches.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the describedembodiments. However, it will be apparent to one skilled in the art thatthe specific details are not required in order to practice the describedembodiments. Thus, the foregoing descriptions of specific embodimentsare presented for purposes of illustration and description. They are notintended to be exhaustive or to limit the described embodiments to theprecise forms disclosed. It will be apparent to one of ordinary skill inthe art that many modifications and variations are possible in view ofthe above teachings. The specification and drawings are, accordingly, tobe regarded in an illustrative rather than restrictive sense.

The various aspects, implementations, or features of the describedembodiments may be used separately or in any combination. Variousaspects of the described embodiments may be implemented by software,hardware or a combination of hardware and software. The describedembodiments may also be embodied as computer readable code on a computerreadable medium for controlling manufacturing operations or as computerreadable code on a computer readable medium for controlling amanufacturing line. The computer readable medium is any data storagedevice that may store data, which may thereafter be read by a computersystem. Examples of the computer readable medium include read-onlymemory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tape, andoptical data storage devices. The computer readable medium may also bedistributed over network-coupled computer systems so that the computerreadable code is stored and executed in a distributed fashion.

Thus, each of the processes, methods, and algorithms described hereinand/or depicted in the figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems mayinclude computers (e.g., servers) programmed with specific computerinstructions or special purpose computers, special purpose circuitry,and so forth. A code module may be compiled and linked into anexecutable program, installed in a dynamic link library, or may bewritten in an interpreted programming language. In some embodiments,particular operations and methods may be performed by circuitry that isspecific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itwill be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

It will be appreciated that the systems and methods of the disclosureeach have several innovative aspects, no single one of which is solelyresponsible or required for the desirable attributes disclosed herein.The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure.

Certain features that are described in this specification in the contextof separate embodiments also may be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also may be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

It will be appreciated that conditional language used herein, such as,among others, “can,” “could,” “might,” “may,” “e.g.,” and the like,unless specifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or withoutauthor input or prompting, whether these features, elements and/or stepsare included or are to be performed in any particular embodiment. Theterms “comprising,” “including,” “having,” and the like are synonymousand are used inclusively, in an open-ended fashion, and do not excludeadditional elements, features, acts, operations, and so forth. Also, theterm “or” is used in its inclusive sense (and not in its exclusivesense) so that when used, for example, to connect a list of elements,the term “or” means one, some, or all of the elements in the list. Inaddition, the articles “a,” “an,” and “the” as used in this applicationand the appended claims are to be construed to mean “one or more” or “atleast one” unless specified otherwise. Similarly, while operations maybe depicted in the drawings in a particular order, it is to berecognized that such operations need not be performed in the particularorder shown or in sequential order, or that all illustrated operationsbe performed, to achieve desirable results. Further, the drawings mayschematically depict one more example processes in the form of aflowchart. However, other operations that are not depicted may beincorporated in the example methods and processes that are schematicallyillustrated. For example, one or more additional operations may beperformed before, after, simultaneously, or between any of theillustrated operations. Additionally, the operations may be rearrangedor reordered in other embodiments. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems may generally be integrated together in a singlesoftware product or packaged into multiple software products.Additionally, other embodiments are within the scope of the followingclaims. In some cases, the actions recited in the claims may beperformed in a different order and still achieve desirable results.

Accordingly, the claims are not intended to be limited to theembodiments shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

What is claimed is:
 1. A method implemented by a display systemcomprising one or more processors, the display system presentingaugmented reality virtual content to a user, and the method comprising:obtaining an image of an eye of the user; determining one or moremeasures associated with motion blur represented in the image;determining, based on the one or more measures, that the image showsperformance of a saccade; and performing, by the display system, one ormore actions in response, the actions being associated with a reductionin visual perceptibility.
 2. The method of claim 1, wherein determiningone or more measures associated with motion blur comprises: extractingfrequency information from the image; and determining a magnitude ofmotion blur based on the frequency information, the magnitude beingutilized as the measure of motion blur.
 3. The method of claim 2,wherein determining one or more measures associated with motion blurcomprises: convolving one or more image kernels with at least a portionof the image; and determining the measure based on the convolution. 4.The method of claim 1, wherein determining one or more measurescomprises determining a plurality of measures, each measure beingassociated with a respective motion blur determination scheme; andwherein determining that the image shows performance of a saccade isbased on the plurality of measures.
 5. The method of claim 1, whereinthe one or more measures are averaged, and wherein determining that theimage shows performance of a saccade comprises determining that theaverage exceeds a predetermined threshold value.
 6. The method of claim1, wherein determining one or more measures comprises determining aplurality of measures, and determining that the image shows performanceof a saccade comprises determining that a threshold number of themeasures exceeds respective predetermined threshold values.
 7. Themethod of claim 6, wherein the threshold number is a majority of themeasures or a user-selectable threshold number.
 8. The method of claim1, further comprising: determining a magnitude and direction associatedwith the motion blur.
 9. The method of claim 8, further comprising:accessing information identifying a three-dimensional fixation pointassociated with the user, the fixation point representing a gaze of theuser; and estimating an updated three-dimensional fixation point basedon the determined magnitude and direction, the updated three-dimensionalfixation point representing a gaze of the user subsequent to thesaccade.
 10. The method of claim 1, wherein the display system furthercomprises: a wearable display configured to present virtual content to auser; and an imaging device, wherein obtaining an image of the eye ofthe user is performed by the imaging device.
 11. The method of claim 1,wherein determining, based on the one or more measures, that the imageshows performance of a saccade comprises determining whether one or moremeasures associated with motion blur is exhibited in one or more regionsof the image.
 12. The method of claim 11, wherein determining whetherone or more measures of motion blur is exhibited in one or more regionsof the image comprises: identifying a target region of the imagecorresponding to a target portion of the user's eye; and determiningwhether the measure or more of motion blur is exhibited in theidentified region of the image.
 13. The method of claim 12, wherein thedisplay system further comprises an infrared light source, wherein thetarget portion of the user's eye comprises an illuminated portion of theuser's eye, wherein the illuminated portion of the user's eye isilluminated by the infrared light source.
 14. The method of claim 12,wherein the target portion of the user's eye comprises one or more eyefeatures selected from the group consisting of a pupil, at least aportion of an iris, and a portion of a limbic boundary.
 15. The methodof claim 11, further comprising: adjusting, based at least in part on adetermination that the image shows performance of a saccade, one or morepower settings of the display system, or one or more virtual contentrendering settings.
 16. The method of claim 11, further comprising:changing, in response to a determination that the image showsperformance of a saccade, a depth plane on which virtual content ispresented.
 17. The method of claim 1, where, the display system furthercomprises a stack of waveguides, wherein one or more of the waveguidesare configured to output light with different amounts of wavefrontdivergence than others of the waveguides.
 18. The method of claim 1,wherein performing one or more actions comprises: determining anaccommodation vergence mismatch; and in response to determining that theaccommodation vergence mismatch exceeds a threshold, change a wavefrontdivergence of the outputted light forming the virtual content.
 19. Themethod of claim 1, wherein performing one or more actions comprisesswitching depth planes of at least a portion of the virtual contentpresented by the display system.
 20. The method of claim 1, whereinperforming one or more actions comprises reducing rendering quality ofat least a portion of the virtual content presented by the displaysystem.